how to limit SCSI TCQ depth in FreeBSD?

[Don Lewis]

On 11 Jun, Frantisek Rysanek wrote:
> Dear SCSI experts and FreeBSD veterans,
> 
> I'm looking for advice with an apparently simple SCSI issue:
> how do I set a maximum TCQ depth in FreeBSD?
> 
> Scenario:
> - an U320 SCSI disk drive
> - Adaptec 39320 (aic7902) == U320 & PCI-X
> - FreeBSD 5.2.1 (and Linux 2.4.26, for comparison)
> - Bonnie (-s 1500, 4x in parallel, to trash the RAM buffers)
> - the system is a dual Xeon @ 2.6 GHz, 1 GB DDR266,
>    ServerWorks GC-LE (PCI-X)
> 
> I have encountered a SCSI disk device that tends to hang
> under load in FreeBSD - namely I have discovered that using
> Bonnie. The disk hangs when Bonnie says "rewriting" and in 
> iostat you can see the TPS skyrocket.
> 
> In Linux, with the default compile-time parameters to 
> aic79xx.o, the disk works without a problem, no matter what 
> Bonnie does to it.
> 
> I figured out that it could be a problem with TCQ depth. The 
> default in Linux/aic79xx is 32, so I cranked it up to 250, 
> recompiled and ran Bonnie. Guess what - the disk hung at just 
> the same point as in FreeBSD, only the TPS in linux managed to 
> crawl yet somewhat higher.
> 
> The hang invokes a flurry of error messages on the host 
> system, both in Linux and FreeBSD - the precise wording is 
> different, but essentially the drivers say that the SCSI bus 
> is stuck and they keep trying to reset it, to no avail. The 
> culprit device remains stuck until it's power-cycled.
> 
> The datasheets for the disk device say that it can cope with
> 256 commands in the TCQ. Yet my experiments show that
> the highest safe value is about 224. 240 already makes the 
> device crash.
> I tried to find the breakdown point by interval halving.
> An increase from 32 to 128 on the host side results in
> a significant performance improvement (about +50 %).
> A further increase to 192 or 224 is not much of an advantage.
> At 240, the disk chokes.
> 
> To sum up, I'm fairly convinced that the disk can run just
> fine at a lower TCQ depth.
> 
> The trouble is, that I can't find a way to set the TCQ depth
> in FreeBSD.

Try the "camcontrol tags" command.

As an alternative, you might want to use the "camcontrol modepage"
command to turn off write caching, especially if you don't want the
drive lie to soft updates about when data has been written to stable
storage.  The file system benchmarking that I"ve done with soft updates
enabled on SCSI drives with TCQ has shown little difference in
performance no matter whether write caching is enabled or not.  The same
drive controller buffer memory is used for the write cache and the TCQ
cache in the drives that I've tested, so when write caching is enabled
and the write cache fills up, the drive drops the number of outstanding
tagged commands that it supports.  I think the only real difference is
when the drive sends back the acknowledgement that the data has been
written.

I have seen cases where a drive fails to service certain requests for an
arbitrarily long period of time when it is saturated with a string of
other (read?) requests that it can satisfy from cache, or at least that
is what I think is going on.  Reducing the TCQ depth and asking the
vendor for a firmware fix were the two suggestions that I got.