Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

[Sten Daniel Soersdal]

Alexander Goncharov wrote:
> Hello world, I have faced with following issue on my dedicated server:
> 
> 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10
> 
> FreeBSD 7.0-generic, 64 bit version
> 
> 
> 
> IO performance and behavior is very strange:
> 
> 1)       No other process are running:
> 
> Memory stat:
> 
> Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free
> 
> Copy 3GB file first time
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec)
> 
> 20MBS is very poor…
> 
> Memory stat now:
> 
> Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free
> 
> 2)       Copy the same file again:
> 
>  dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec)
> 
> 100MBs – much better
> 
> Mem: 9048K Active, 5951M Inact, 287M Wired, 32K Cache, 214M Buf, 25G Free
> 
> Next attempts with this file show 100MBs spped
> 
> 3)Copy other file first time
> dd if=/home/test2 of=/home/test2_2
> 6144000+0 records in
> 6144000+0 records out
> 3145728000 bytes transferred in 141.870921 secs (22173170 bytes/sec)
> 
> 20MBs again
> 
> Copy the same file again:
> 
> dd if=/home/test2 of=/home/test2_2
> 6144000+0 records in
> 6144000+0 records out
> 3145728000 bytes transferred in 29.560267 secs (106417441 bytes/sec)
> 
> 100MBs – much better. Memory stat:
> 
>  Mem: 8964K Active, 12G Inact, 287M Wired, 28K Cache, 214M Buf, 19G Free
> 
> Copy first file again:
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 34.310753 secs (93883847 bytes/sec)
> 
> good speed
> 
> ....
> 
> So, first copying of any file is extremely slow, second and all next
> attempts  are much better. Please help me with this issue.
> 
> Dmesg  some info:
> 
> 
> ioapic0 <Version 1.1> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
> hptrr: HPT RocketRAID controller driver v1.1 (May 22 2008 00:49:14)
> acpi0: <PTLTD   RSDT> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x8008-0x800b on acpi0
> 
> 
> amr0: <LSILogic MegaRAID 1.53> mem
> 0xdc400000-0xdc40ffff,0xdc100000-0xdc13ffff irq 16 at device 14.0 on pci140
> amr0: Using 64-bit DMA
> amr0: [ITHREAD]
> amr0: delete logical drives supported by controller
> amr0: <LSILogic MegaRAID SCSI 320-2E> Firmware 514L, BIOS H430, 128MB RAM

I'm not sure if it's relevant but the LSILogic MegaRAID cards i have are 
not able to do true RAID 1+0. Instead it mirrors and then concatenates 
which will leave you relatively poor read/write performance (no 
striping). To do RAID 1+0 on my controllers i had to do mirror in BIOS 
and then stripe the mirror's using gstripe.

Also once you do RAID you have to pay great attention to block 
sizes/widths/etc and alignment of such. Improper alignment will ruin any 
performance benefits your RAID levels may provide.

Without Battery backed cache your write performance will be poor.

Also 'dd' defaults to 512 byte blocks (poor testing).

Just my $0.2

-- 
Sten Daniel Soersdal