ZFS read performance

Thu Jan 30 17:45:31 UTC 2014

Hi Steve,

At this point, I am just doing some evaluation with what's on-hand.  Ultimately I would
add RAM and redundancy, etc.

bs was definitely a big part of the issue.  (*forehead slap*)

hw-mfi.max_cmds=-1 also make a significant impact.

Onwards to some more strenuous testing...

Thanks for the help!

-Markham

On 14-01-28 6:34 PM, Steven Hartland wrote:
> First off avoid using mfi its a RAID card which adds a extra layer
> you dont want even in jbob mode.
>
> Whats the mfi max_cmd set to? If you havent already set it to -1
> /boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max.
>
> Next 4GB on a machine avoid it if possible, thats whats disabling
> prefetch by default, I'd recommend a min of 8Gb more if posslbe.
>
> When you "add" a disk your creating a stripe which is bad news if
> any single disk fails, so avoid that at all costs.
>
> Finally your issue with read performance is likely your block size,
> you set it for write to 1mb (which is reasonable) but you didn't
> on read so will be 512b which will likely cpu limit in the dd
> process instead of bottlenecking on the disk / FS performance.
>
>    Regards
>    Steve
> ----- Original Message ----- From: "markham breitbach" <markham_breitbach at ssimicro.com>
> To: <freebsd-performance at freebsd.org>
> Sent: Wednesday, January 29, 2014 12:02 AM
> Subject: ZFS read performance
>
>
>>   Hi,
>>   I'm trying to figure out a ZFS read performance issue that I am seeing
>>   on FreeBSD9.2 (amd64).
>>   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
>>   K8-class CPU)
>>   real memory  = 4294967296 (4096 MB)
>>   avail memory = 4059762688 (3871 MB)
>>   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
>>   with the drives installed JBOD.  These are intended for use as a
>>   storage pool.  The main system drives are using the onboard SATA
>>   controller for the system.
>>
>>   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
>>   Write speeds using dd as follows:
>>   WRITE:
>>   dd if=/dev/zero of=/vol0/test bs=1m count=10000
>>   READ
>>   dd if=/vol0/test of=/dev/null
>>   When I create a zfs pool with a single drive:
>>   zpool create vol0 /dev/mfisyspd0
>>   using dd again, I can write ~175MBps, but my read speed is only about
>>   60MBps.
>>   I have set
>>   vfs.zfs.prefetch_disable = 0
>>   and this improves reads to about 70MBps, but no much beyond that.
>>   Now, this gets even more interesting.
>>   If I add a drive to the pool:
>>   zpool add vol0 /dev/mfisyspd1
>>   My write speeds increase to ~280MBps, but my read speed is still about
>>   60MBps.
>>   Using gstat, I can see that the load is shared pretty equally amongst
>>   the drives.
>>   This trend continues with the addition of a 3rd and 4th drive with
>>   write speeds ramping up to about 500MBps (I didn't test past that), but
>>   read speeds stuck around 60MBps.
>>   I have done similar testing under 9.1 making sure to align 4K sectors
>>   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
>>   I have looked through the wiki and handbook for tuning and done some
>>   googling around, but I'm pretty much out of ideas now.  Everything I
>>   have read seems to indicate that ZFS should run fine on 4GB without any
>>   tuning, and at this point I haven't even begun any serious stress
>>   testing.
>>   Any ideas of where to go from here would be greatly appreciated.
>>   Thanks,
>>   -Markham
>>
>>   ---
>>   [1]Markham Breitbach
>>   Network Operations
>>   SSi   People, Ideas, Technology
>>   - - - - - - - - - - - - - - - - - - - - -
>>   +1 867 669 7500 work
>>   +1 867 669 7510 fax
>>   [2]markham_breitbach at ssimicro.com
>>   [3]www.ssimicro.com
>>   356B Old Airport Road
>>   Yellowknife , NT X1A 3T4
>>   Canada
>>   - - - - - - - - - - - - - - - - - - - - -
>>   Visit some of our other networks
>>   [4]www.qiniq.com   &   [5]www.airware.ca
>>
>> References
>>
>>   1. http://www.ssimicro.com/
>>   2. mailto:markham_breitbach at ssimicro.com
>>   3. http://www.ssimicro.com/
>>   4. http://www.qiniq.com/
>>   5. http://www.airware.ca/
>> _______________________________________________
>> freebsd-performance at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
>>
>
> ================================================
> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or
> entity to whom it is addressed. In the event of misdirection, the recipient is
> prohibited from using, copying, printing or otherwise disseminating it or any
> information contained in it.
> In the event of misdirection, illegible or incomplete transmission please telephone +44
> 845 868 1337
> or return the E.mail to postmaster at multiplay.co.uk.
>