Frequent disk I/O stalls while building (poudriere), processes in "zfs tear" state

[Felix Palmen]

* Dewayne Geraghty <dewayne at heuristicsystems.com.au> [20210416 06:26]:
> On 16/04/2021 2:29 am, Felix Palmen wrote:
> > Right now, I'm running a test with idprio 0 instead, which still seems
> > to have the desired effect, and so far, I didn't have any of these
> > stalls. If this persists, the problem is solved for me!
> > 
> > I'd still be curious about what might be the cause, and, what this state
> > "zfs tear" actually means. But that's kind of an "academic interest"
> > now.
> 
> Most likely your other processes are pre-empting your build, which is
> what you want :).

Yes, that's exactly the plan.

> Use /usr/bin/top to see the priority of the processes (ie under the  PRI
> column).  Using an idprio 22, means (on my 12.2Stable) a PRI of 146.  If
> your kern.sched.preempt_thresh is using the default (of 80) then
> processes with a PRI of <80 will preempt (for IO).

I was doing that a lot, that's how I found those "global" I/O stalls
were happening when some processes were in that "zfs tear" state (shown
in top only as "zfs te").

> Even with an idprio 0, the PRI is 124. So I suspect that was more a
> matter of timing (ie good luck).

That seems kind of unlikely because the behavior is pretty reproducible.
Having observed builds on idprio 0 (yes, this results in a priority of
124) for a while, I still see from time to time processes getting
"stuck" for a few seconds, mostly ccache processes, but now in state
"zfsvfs" and the rest of the system is not affected, I/O still works.

So, something did change with ZFS and priorities between 12.2 and 13.0.
Running the whole builds on idprio 22 worked fine on 12.2.

> You could increase your pre-emption threshold for the duration of the
> build, to include your nice value. But... (not really a good idea).

That would clearly defeat the purpose, yes ;)

> Re zfs - sorry, I'm peculiar and don't use it ;)

I suspect the relevant change to be exactly in that context, still
thanks for answering :) Now that I have a working solution, it isn't an
important issue for me any more. Curiosity remains…

-- 
 Dipl.-Inform. Felix Palmen  <felix at palmen-it.de>   ,.//..........
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key}     http://palmen-it.de/pub.txt   //   """""""""""
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20210416/d528557a/attachment.sig>