how to limit SCSI TCQ depth in FreeBSD?

[Frantisek Rysanek]

Dear SCSI experts and FreeBSD veterans,

I'm looking for advice with an apparently simple SCSI issue:
how do I set a maximum TCQ depth in FreeBSD?

Scenario:
- an U320 SCSI disk drive
- Adaptec 39320 (aic7902) == U320 & PCI-X
- FreeBSD 5.2.1 (and Linux 2.4.26, for comparison)
- Bonnie (-s 1500, 4x in parallel, to trash the RAM buffers)
- the system is a dual Xeon @ 2.6 GHz, 1 GB DDR266,
   ServerWorks GC-LE (PCI-X)

I have encountered a SCSI disk device that tends to hang
under load in FreeBSD - namely I have discovered that using
Bonnie. The disk hangs when Bonnie says "rewriting" and in 
iostat you can see the TPS skyrocket.

In Linux, with the default compile-time parameters to 
aic79xx.o, the disk works without a problem, no matter what 
Bonnie does to it.

I figured out that it could be a problem with TCQ depth. The 
default in Linux/aic79xx is 32, so I cranked it up to 250, 
recompiled and ran Bonnie. Guess what - the disk hung at just 
the same point as in FreeBSD, only the TPS in linux managed to 
crawl yet somewhat higher.

The hang invokes a flurry of error messages on the host 
system, both in Linux and FreeBSD - the precise wording is 
different, but essentially the drivers say that the SCSI bus 
is stuck and they keep trying to reset it, to no avail. The 
culprit device remains stuck until it's power-cycled.

The datasheets for the disk device say that it can cope with
256 commands in the TCQ. Yet my experiments show that
the highest safe value is about 224. 240 already makes the 
device crash.
I tried to find the breakdown point by interval halving.
An increase from 32 to 128 on the host side results in
a significant performance improvement (about +50 %).
A further increase to 192 or 224 is not much of an advantage.
At 240, the disk chokes.

To sum up, I'm fairly convinced that the disk can run just
fine at a lower TCQ depth.

The trouble is, that I can't find a way to set the TCQ depth
in FreeBSD.

Although the directory structure seems similar, the FreeBSD
drivers for aic7xxx/aic79xx are significantly different from 
the Linux variety. The preprocessor macros and structs that 
handle TCQ settings in Linux are clearly missing in FreeBSD.
I understand that FreeBSD is using a common SCSI and CAM
layer, so perhaps I'd better be looking somewhere in there - 
only I can't find a hint of what to do. I tried looking into
/usr/src/sys/conf/options
/usr/src/sys/conf/NOTES
/usr/src/sys/dev/aic7xxx/*
etc. to no avail. I'm not too much of a coder, I get lost 
quickly in the CAM and SCSI layers.

Any ideas are welcome.
Thanks for your time reading this message :-)

Frank Rysanek