performance with LSI SAS 1064

[Eric Anderson]

Scott Long wrote:
> Eric Anderson wrote:
>> Lutieri G. wrote:
>>> I've make a test with dd command:
>>>
>>> # time dd if=/dev/zero of=./8gbfile bs=1024k count=8192
>>> 8192+0 records in
>>> 8192+0 records out
>>> 8589934592 bytes transferred in 155.653213 secs (55186362 bytes/sec)
>>> 0.007u 25.129s 2:35.69 16.1%    55+6039k 117+68628io 0pf+0w
>>>
>>> in other terminal i ran iostat while dd were running and I get this:
>>>
>>> # iostat -I 1
>>>       tty             da0            pass0             cpu
>>>  tin tout  KB/t xfrs   MB   KB/t xfrs   MB  us ni sy in id
>>>    1   61 25.23 52958 1305.07   0.00   0  0.00   0  0  8  0 91
>>>    0  184 127.48 434 54.03   0.00   0  0.00   0  0  5  0 95
>>>    0   61 127.49 440 54.78   0.00   0  0.00   0  0  5  0 95
>>>    0   61 127.75 445 55.52   0.00   0  0.00   0  0  5  0 95
>>>    0   61 127.49 442 55.03   0.00   0  0.00   0  0  4  0 96
>>>    0   61 127.49 436 54.28   0.00   0  0.00   0  0  5  0 95
>>>    0   61 125.27 425 51.99   0.00   0  0.00   0  0  5  1 94
>>>    0   61 118.14 393 45.34   0.00   0  0.00   0  0  3  0 97
>>>
>>> average  54MB/s with or without hw.mpt.enable_sata_wc seted in 
>>> loader.conf file.
>>>
>>> is it a normal speed for this adapter?!
>>>
>>>
>>
>>
>> I'm confused - you said in your first post you were getting 3MB/s, 
>> where  above you show something like 55MB/s.
>>
>> You didn't say what kind of disks, or how many, the configuration, etc 
>> - so it's hard to answer much.  The 55MB/s seems pretty decent for 
>> many hard drives in a sequential use state (which is what dd tests 
>> really).
>>
>> Your errors before were probably caused because your queue depth is 
>> set to 255 (or 256?) and the adapter can't do that many.  You should 
>> use camcontrol to reduce it, to maybe 32.  See the camcontrol man page 
>> for the right usage.  It's something that needs setting on every boot, 
>> so a startup file is a good place for it maybe.
>>
>> Eric
>>
>>
> 
> Well, if he's using SATA (which I kinda assumed originally without
> asking) then queue depth isn't going to matter; the MPT driver has no
> interaction with how SATA NCQ operates, if he even has a rev of the
> LSI chip that supports NCQ at all.  If he's using SAS, then queue
> depth will only be a minor factor, CAM is pretty good at autosizing
> the depth with minimal impact.  Now, if he's using SAS disks then
> the boot tunable that I gave him will indeed have no impact at all.
> 
> I believe that the Sun 4100 uses 2.5" disks, whether SATA or SAS.
> 54MB/s is not all that bad for disks of this size.  It's pretty close
> to what I would expect, actually.

If he's using a SAS Seagate 15k rpm 2.5" drive, he could see much better 
than 55MB/s.  I have an LSI (PCI-X 133, model 1064 as he does) with some 
Seagate 15k RPM drives, and I can get 100MB/s.  Tests were done on 
FreeBSD 7-CURRENT, with write caching enabled on the drive. Here's some 
of the numbers:

    1k block read size:    6390033 bytes/sec
    2k block read size:   12257594 bytes/sec
    4k block read size:   23071775 bytes/sec
    8k block read size:   41858586 bytes/sec
   16k block read size:   71242623 bytes/sec
   32k block read size:  108019212 bytes/sec
   64k block read size:  108043811 bytes/sec
  128k block read size:  108881295 bytes/sec
  256k block read size:  108509827 bytes/sec
  512k block read size:  108670186 bytes/sec
1024k block read size:  108553724 bytes/sec
    1k block write size:   6199645 bytes/sec
    2k block write size:  12169083 bytes/sec
    4k block write size:  23246388 bytes/sec
    8k block write size:  42133751 bytes/sec
   16k block write size:  70968273 bytes/sec
   32k block write size: 108306742 bytes/sec
   64k block write size: 108142985 bytes/sec
  128k block write size: 109156720 bytes/sec
  256k block write size: 109164252 bytes/sec
  512k block write size: 109795665 bytes/sec
1024k block write size: 109202660 bytes/sec


Doing latency tests on /dev/da0
/dev/da0
	512         	# sectorsize
	73407820800 	# mediasize in bytes (68G)
	143374650   	# mediasize in sectors
	8924        	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	63          	# Sectors according to firmware.

Seek times:
	Full stroke:	  250 iter in   1.943102 sec =    7.772 msec
	Half stroke:	  250 iter in   1.567395 sec =    6.270 msec
	Quarter stroke:	  500 iter in   1.671074 sec =    3.342 msec
	Short forward:	  400 iter in   1.159763 sec =    2.899 msec
	Short backward:	  400 iter in   1.014000 sec =    2.535 msec
	Seq outer:	 2048 iter in   0.322013 sec =    0.157 msec
	Seq inner:	 2048 iter in   0.493810 sec =    0.241 msec
Transfer rates:
	outside:       102400 kbytes in   0.968299 sec =   105752 kbytes/sec
	middle:        102400 kbytes in   1.088158 sec =    94104 kbytes/sec
	inside:        102400 kbytes in   1.361127 sec =    75232 kbytes/sec

Doing read tests on /dev/da0
READ: 1k block size:
102400+0 records in
102400+0 records out
104857600 bytes transferred in 16.409555 secs (6390033 bytes/sec)
READ: 2k block size:
51200+0 records in
51200+0 records out
104857600 bytes transferred in 8.554501 secs (12257594 bytes/sec)
READ: 4k block size:
25600+0 records in
25600+0 records out
104857600 bytes transferred in 4.544843 secs (23071775 bytes/sec)
READ: 8k block size:
12800+0 records in
12800+0 records out
104857600 bytes transferred in 2.505044 secs (41858586 bytes/sec)
READ: 16k block size:
6400+0 records in
6400+0 records out
104857600 bytes transferred in 1.471838 secs (71242623 bytes/sec)
READ: 32k block size:
3200+0 records in
3200+0 records out
104857600 bytes transferred in 0.970731 secs (108019212 bytes/sec)
READ: 64k block size:
1600+0 records in
1600+0 records out
104857600 bytes transferred in 0.970510 secs (108043811 bytes/sec)
READ: 128k block size:
800+0 records in
800+0 records out
104857600 bytes transferred in 0.963045 secs (108881295 bytes/sec)
READ: 256k block size:
400+0 records in
400+0 records out
104857600 bytes transferred in 0.966342 secs (108509827 bytes/sec)
READ: 512k block size:
200+0 records in
200+0 records out
104857600 bytes transferred in 0.964916 secs (108670186 bytes/sec)
READ: 1024k block size:
100+0 records in
100+0 records out
104857600 bytes transferred in 0.965951 secs (108553724 bytes/sec)
Doing write tests on /dev/da0
WRITE: 1k block size:
102400+0 records in
102400+0 records out
104857600 bytes transferred in 16.913484 secs (6199645 bytes/sec)
WRITE: 2k block size:
51200+0 records in
51200+0 records out
104857600 bytes transferred in 8.616722 secs (12169083 bytes/sec)
WRITE: 4k block size:
25600+0 records in
25600+0 records out
104857600 bytes transferred in 4.510705 secs (23246388 bytes/sec)
WRITE: 8k block size:
12800+0 records in
12800+0 records out
104857600 bytes transferred in 2.488684 secs (42133751 bytes/sec)
WRITE: 16k block size:
6400+0 records in
6400+0 records out
104857600 bytes transferred in 1.477528 secs (70968273 bytes/sec)
WRITE: 32k block size:
3200+0 records in
3200+0 records out
104857600 bytes transferred in 0.968154 secs (108306742 bytes/sec)
WRITE: 64k block size:
1600+0 records in
1600+0 records out
104857600 bytes transferred in 0.969620 secs (108142985 bytes/sec)
WRITE: 128k block size:
800+0 records in
800+0 records out
104857600 bytes transferred in 0.960615 secs (109156720 bytes/sec)
WRITE: 256k block size:
400+0 records in
400+0 records out
104857600 bytes transferred in 0.960549 secs (109164252 bytes/sec)
WRITE: 512k block size:
200+0 records in
200+0 records out
104857600 bytes transferred in 0.955025 secs (109795665 bytes/sec)
WRITE: 1024k block size:
100+0 records in
100+0 records out
104857600 bytes transferred in 0.960211 secs (109202660 bytes/sec)
Tests complete.


Eric