access to hard drives is "blocked" by writes to a flash drive

[Don Lewis]

On  3 Mar, Poul-Henning Kamp wrote:

> For various reasons (see: Lemming-syncer) FreeBSD will block all I/O
> traffic to other disks too, when these pileups gets too bad.

The Lemming-syncer problem should have mostly been fixed by 231160 in
head (231952 in stable/9 and 231967 in stable/8) a little over a year
ago. The exceptions are atime updates, mmaped files with dirty pages,
and quotas. Under certain workloads I still notice periodic bursts of
seek noise. After thinking about it for a bit, I suspect that it could
be atime updates, but I haven't tried to confirm that.

When using TCQ or NCQ, perhaps we should limit the number of outstanding
writes per device to leave some slots open for reads.  We should
probably also prioritize reads over writes unless we are under memory
pressure.