FreeBSD -CURRENT regression (only 3.3 MBps on ahc)

[Attila Nagy]

Hello,

I have a server, which has two SCSI controllers, one (two channels) for 
the inner disks and one for a directly attached storage (a Promise 
RM8000 box, which has 8 ATA drives).

Because I use ZFS on the RM8000, I've switched to -CURRENT some time ago 
and noticed that the array's speed fell sharply.

Until now I didn't have the time to investigate it further, but now I 
found out the following:
- in the controller's BIOS the device is recognized as U160 (16 bit 
wide, 80 MHz)
- in dmesg the device can be seen as:
ahc2: <Adaptec 29160 Ultra160 SCSI adapter> port 0x2800-0x28ff mem 
0xfea90000-0xfea90fff irq 31 at device 8.0 on pci1
da2 at ahc2 bus 0 target 0 lun 0
da2: <Promise 8 Disk RAID0 1.10> Fixed Direct Access SCSI-3 device
da2: 3.300MB/s transfers
da2: 1525878MB (3124999168 512 byte sectors: 255H 63S/T 194522C)

The inner disks are OK:
da0: 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)

When I issue a dd, I get exactly 3.3 MBps transfer rates:
dd if=/dev/da2 of=/dev/null bs=1M
^C33+0 records in
33+0 records out
34603008 bytes transferred in 10.130446 secs (3415744 bytes/sec)

Which is worse than that is a parallel read to eight (array as a JBOD, 
each disk as a RAID0 array) disks:
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    1      3      3    383  323.9      0      0    0.0   96.9| da2
    1      3      3    383  323.9      0      0    0.0   96.9| da3
    1      3      3    383  323.9      0      0    0.0   96.9| da4
    1      3      3    383  323.9      0      0    0.0   96.9| da5
    1      3      3    383  323.7      0      0    0.0   96.8| da6
    1      3      3    383  323.9      0      0    0.0   96.9| da7
    1      3      3    383  323.9      0      0    0.0   96.9| da8
    1      3      3    383  323.8      0      0    0.0   96.8| da9

It makes the array pretty useless...

However, when I do a:
camcontrol negotiate da2 -a -W 16
Current Parameters:
(pass3:ahc2:0:0:0): sync parameter: 0
(pass3:ahc2:0:0:0): offset: 0
(pass3:ahc2:0:0:0): bus width: 8 bits
(pass3:ahc2:0:0:0): disconnection is disabled
(pass3:ahc2:0:0:0): tagged queueing is disabled
New Parameters:
(pass3:ahc2:0:0:0): sync parameter: 0
(pass3:ahc2:0:0:0): offset: 0
(pass3:ahc2:0:0:0): bus width: 16 bits
(pass3:ahc2:0:0:0): disconnection is disabled
(pass3:ahc2:0:0:0): tagged queueing is disabled

(TCQ, disconnection is disabled in the BIOS, because that was my first idea)

After this, I get:
dd if=/dev/da2 of=/dev/null bs=1M
^C76+0 records in
76+0 records out
79691776 bytes transferred in 11.871042 secs (6713124 bytes/sec)

and:
camcontrol inquiry da2
pass3: <Promise 8 Disk RAID0 1.10> Fixed Direct Access SCSI-3 device
pass3: Serial Number                
pass3: 6.600MB/s transfers  (16bit)

The transfer rate has doubled.

I have tried to do a:
camcontrol negotiate da2 -a -R 80 -W 16
Current Parameters:
(pass3:ahc2:0:0:0): sync parameter: 0
(pass3:ahc2:0:0:0): offset: 0
(pass3:ahc2:0:0:0): bus width: 16 bits
(pass3:ahc2:0:0:0): disconnection is disabled
(pass3:ahc2:0:0:0): tagged queueing is disabled
New Parameters:
(pass3:ahc2:0:0:0): sync parameter: 0
(pass3:ahc2:0:0:0): offset: 0
(pass3:ahc2:0:0:0): bus width: 16 bits
(pass3:ahc2:0:0:0): disconnection is disabled
(pass3:ahc2:0:0:0): tagged queueing is disabled

but without any effect.

I can't show a dmesg now from the previous kernels, but this server 
could achieve 20-30 MBps from that array with FreeBSD 5,6 (when it went 
out of production) and certainly not just 3.3 MBps with 7-STABLE until a 
point.
I have had a: @(#)FreeBSD 7.0-STABLE #4: Mon Jun  9 12:21:13 CEST 2008
kernel which worked reasonably well (performance wise, ZFS was unstable) 
and I first noticed the slowdowns when I've upgraded from that, first in 
the line of STABLE, then to CURRENT.

Any ideas about that?

Thanks,