Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

[Alexander Goncharov]

Hello world, I have faced with following issue on my dedicated server:

8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10

FreeBSD 7.0-generic, 64 bit version



IO performance and behavior is very strange:

1)       No other process are running:

Memory stat:

Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free

Copy 3GB file first time

dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec)

20MBS is very poor…

Memory stat now:

Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free

2)       Copy the same file again:

 dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec)

100MBs – much better

Mem: 9048K Active, 5951M Inact, 287M Wired, 32K Cache, 214M Buf, 25G Free

Next attempts with this file show 100MBs spped

3)Copy other file first time
dd if=/home/test2 of=/home/test2_2
6144000+0 records in
6144000+0 records out
3145728000 bytes transferred in 141.870921 secs (22173170 bytes/sec)

20MBs again

Copy the same file again:

dd if=/home/test2 of=/home/test2_2
6144000+0 records in
6144000+0 records out
3145728000 bytes transferred in 29.560267 secs (106417441 bytes/sec)

100MBs – much better. Memory stat:

 Mem: 8964K Active, 12G Inact, 287M Wired, 28K Cache, 214M Buf, 19G Free

Copy first file again:

dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 34.310753 secs (93883847 bytes/sec)

good speed

....

So, first copying of any file is extremely slow, second and all next
attempts  are much better. Please help me with this issue.

Dmesg  some info:


ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (May 22 2008 00:49:14)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x8008-0x800b on acpi0


amr0: <LSILogic MegaRAID 1.53> mem
0xdc400000-0xdc40ffff,0xdc100000-0xdc13ffff irq 16 at device 14.0 on pci140
amr0: Using 64-bit DMA
amr0: [ITHREAD]
amr0: delete logical drives supported by controller
amr0: <LSILogic MegaRAID SCSI 320-2E> Firmware 514L, BIOS H430, 128MB RAM


hptrr: no controller detected.
acd0: DVDROM <DV-28E-R/1.8A> at ata1-slave UDMA33
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 137328MB (281247744 sectors) RAID 1 (optimal)


Best regards,

Alexander.