Extremely low disk throughput under high compute load

[Stefan Esser]

My i7-2600K based system with 24 GB RAM was in the midst of a buildworld -j8
(starting from a clean state) which caused a load average of 12 for more than
1 hour, when I decided to move a directory structure holding some 10 GB to its
own ZFS file system. File sizes varied, but were mostly in the range 0f 500KB.

I had just thrown away /usr/obj, but /usr/src was cached in ARC and thus there
was nearly no disk activity caused by the buildworld.

The copying proceeded at a rate of at most 10 MB/s, but most of the time less
than 100 KB/s were transferred. The "cp" process had a PRIO of 20 and thus a
much better priority than the compute bound compiler processes, but it got
just 0.2% to 0.5% of 1 CPU core. Apparently, the copy process was scheduled
at such a low rate, that it only managed to issue a few controller writes per
second.

The system is healthy and does not show any problems or anomalies under
normal use (e.g., file copies are fast, without the high compute load).

This was with SCHED_ULE on a -CURRENT without WITNESS or malloc debugging.

Is this a regression in -CURRENT?

Regards, STefan