Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

[Gary Palmer]

On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander Goncharov wrote:
> Hello world, I have faced with following issue on my dedicated server:
> 
> 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10
> 
> FreeBSD 7.0-generic, 64 bit version
> 
> 
> 
> IO performance and behavior is very strange:
> 
> 1)       No other process are running:
> 
> Memory stat:
> 
> Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free
> 
> Copy 3GB file first time
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec)
> 
> 20MBS is very poor?
> 
> Memory stat now:
> 
> Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free
> 
> 2)       Copy the same file again:
> 
>  dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec)
> 
> 100MBs ? much better
> 

The "Inact" (Inactive) went up - the 3GiB file is now cached in 
memory.  So the second (and subsequent) runs are going from cached
memory so your 100MiB/sec transfer is actually just testing write
speed, not read/write speed.  This is the same for your other
tests too

"Inact" is memory that has been used and is being kept around incase
it is used again, in other words its caching file data in the
"Inact" region in top.  It'll be reused if something else needs
the memory, but until then it sticks around.

Remember - you are copying the file from and to the same
filesystem - this is always going to appear slow relative to
pure read or pure write tests.  A pure write test is effectively
what you have when you're getting your 100MiB/sec test result since
its just writing out from cache memory.

Regards,

Gary