ZFS performance question

[Heinrich Rebehn]

On 20.08.2010, at 12:48, Olivier Smedts wrote:

> 2010/8/20 Heinrich Rebehn <rebehn at ant.uni-bremen.de>:
>> Hi all,
>> 
>> After setting up our new server (Intel Q9550 CPU, 8GB RAM, 4 x ST31000340NS) i did a bonnie++ benchmark on the zfs raidz that i created on 4 partitions on the 4 disks.
>> 
>> root at antsrv4 [/data/nocompression] # bonnie++ -u root -d .
>> Using uid:0, gid:0.
>> Writing a byte at a time...done
>> Writing intelligently...done
>> Rewriting...done
>> Reading a byte at a time...done
>> Reading intelligently...done
>> start 'em...done...done...done...done...done...
>> Create files in sequential order...done.
>> Stat files in sequential order...done.
>> Delete files in sequential order...done.
>> Create files in random order...done.
>> Stat files in random order...done.
>> Delete files in random order...done.
>> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>> antsrv4.ant.uni 16G    94  81 151823  39 100780  27   303  99 239853  30 148.3   4
>> Latency              4733ms    6051ms    8173ms   37504us     917ms    1030ms
>> Version  1.96       ------Sequential Create------ --------Random Create--------
>> antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>>              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>                 16 24183  97 +++++ +++ 21715  98 17209  98 +++++ +++ 21291  97
>> Latency             19779us     128us     162us   39671us      38us      77us
>> 1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282303232,16G,,94,81,151823,39,100780,27,303,99,239853,30,148.3,4,16,,,,,24183,97,+++++,+++,21715,98,17209,98,+++++,+++,21291,97, 733ms,6051ms,8173ms,37504us,917ms,1030ms,19779us,128us,162us,39671us,38us,77us
>> 
>> 
>> I am somewhat concerned about the numbers for per-char-output and per-char-input. In fact, i have never before seen that low numbers in a bonnie test. Using a single disk with UFS yields about 6 times as much.
>> 
>> 
>> I know that this is not crucial for a normal file server, but i want to rule out any configuration errors.
>> 
>> Is this normal for ZFS?. Disks are accessed using ahci. I did not attempt any tuning yet.
> 
> Can you try without AHCI ?
> http://www.phoronix.com/scan.php?page=article&item=freebsd_zfs_cam&num=5

Hmm, that looks impressive, but:

root at antsrv4 [/data/nocompression] # bonnie++ -u root -d .
Using uid:0, gid:0.
[snip]
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
antsrv4.ant.uni 16G   107  99 152759  39 118765  33   304  99 227534  29 137.4   4
Latency               216ms    6110ms    6436ms     108ms     717ms     784ms
Version  1.96       ------Sequential Create------ --------Random Create--------
antsrv4.ant.uni-bre -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23264  94 +++++ +++ 20417  96 24430  97 +++++ +++ 20866  97
Latency             20142us     138us    3506us   19454us     128us      77us
1.96,1.96,antsrv4.ant.uni-bremen.de,1,1282317527,16G,,107,99,152759,39,118765,33,304,99,227534,29,137.4,4,16,,,,,23264,94,+++++,+++,20417,96,24430,97,+++++,+++,20866,97,216ms,6110ms,6436ms,108ms,717ms,784ms,20142us,138us,3506us,19454us,128us,77us

Almost no difference. Only the create rate increased.

root at antsrv4 [/data/nocompression] # kldstat
Id Refs Address            Size     Name
 1    9 0xffffffff80100000 d6aa98   kernel
 2    1 0xffffffff80e6b000 19eb18   zfs.ko
 3    2 0xffffffff8100a000 3868     opensolaris.ko
root at antsrv4 [/data/nocompression] # ls /dev/ada*
ls: /dev/ada*: No such file or directory

-Heinrich
> 
>> BTW: Running OpenSolaris on the same hardware yields 110306 for per-char-write and 94698 for per-char-read.
>> 
> 
> 
> 
> -- 
> Olivier Smedts                                                 _
>                                         ASCII ribbon campaign ( )
> e-mail: olivier at gid0.org        - against HTML email & vCards  X
> www: http://www.gid0.org    - against proprietary attachments / \
> 
>   "Il y a seulement 10 sortes de gens dans le monde :
>   ceux qui comprennent le binaire,
>   et ceux qui ne le comprennent pas."
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"


Heinrich Rebehn

University of Bremen
Physics / Electrical and Electronics Engineering
- Department of Telecommunications -

Phone : +49/421/218-62394
Fax   :            -3341