Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

[Oleg Sharoiko]

Hi!

Alexander Goncharov wrote:
> On Tue, May 27, 2008 at 1:10 AM, Gary Palmer <gpalmer at freebsd.org> wrote:
>
>   
>> On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander Goncharov wrote:
>>     
>>> Hello world, I have faced with following issue on my dedicated server:
>>>
>>> 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10
>>>
>>> FreeBSD 7.0-generic, 64 bit version
>>>
>>>
>>>
>>> IO performance and behavior is very strange:
>>>
>>> 1)       No other process are running:
>>>
>>> Memory stat:
>>>
>>> Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free
>>>
>>> Copy 3GB file first time
>>>
>>> dd if=/home/3gb_file of=/home/3gb_file2
>>> 6291456+0 records in
>>> 6291456+0 records out
>>> 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec)
>>>
>>> 20MBS is very poor?
>>>
>>>       

Alexander, you're using default block size of 512 bytes. In this case 
your hardware has to:

while (!eof(source_file)) {
  locate next 512 byte block
  read it
  locate writing position
  write block
}

So it does 6291456*2 searches. Moreover you're working with files and 
thus dd has to go throug filesystem layer which involves additional 
overhead.

Please try increasing block size to, for example, 8 mb (add bs=8m to the 
arguments of dd)

>>> Memory stat now:
>>>
>>> Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free
>>>
>>> 2)       Copy the same file again:
>>>
>>>  dd if=/home/3gb_file of=/home/3gb_file2
>>> 6291456+0 records in
>>> 6291456+0 records out
>>> 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec)
>>>
>>> 100MBs ? much better
>>>
>>>       
>> The "Inact" (Inactive) went up - the 3GiB file is now cached in
>> memory.  So the second (and subsequent) runs are going from cached
>> memory so your 100MiB/sec transfer is actually just testing write
>> speed, not read/write speed.  This is the same for your other
>> tests too
>>
>> "Inact" is memory that has been used and is being kept around incase
>> it is used again, in other words its caching file data in the
>> "Inact" region in top.  It'll be reused if something else needs
>> the memory, but until then it sticks around.
>>
>> Remember - you are copying the file from and to the same
>> filesystem - this is always going to appear slow relative to
>> pure read or pure write tests.  A pure write test is effectively
>> what you have when you're getting your 100MiB/sec test result since
>> its just writing out from cache memory.
>>
>> Regards,
>>
>> Gary
>>
>>     
>
> Hi Gary,
>
> Big thanks for your quick reply. Read only and write only speed is good
> ~100MBs. But I am not sure if it really good for hw RAID 10 8x 15k rpm
> drives.
> I am worried about read/write speed usecase which is most used at real tasks
> (data base). I was surprised 20 MBs speed, this value is likely to one drive
> speed. But I have 8 high speed drives. Something isn't right here.
>
> I am waiting for freebsd community help.
>
> Thanks in advance.
> Alexander
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
>