Re: qemu-user-static aarch64 lockup/race? (was Re: Python failure in poudriere on arm64 (via qemu-user-static cross compiling))

On 28/01/24 22:23, Warner Losh wrote:
> 
> 
> On Sun, Jan 28, 2024, 12:38 PM Guido Falsi <mad@madpilot.net 
> <mailto:mad@madpilot.net>> wrote:
> 
>     On 28/01/24 15:15, Guido Falsi wrote:
>      > Hi all, again,
>      >
>      > I have some more findings about this, I'm top posting because the
>     old
>      > message is not really that much relevant anymore.
>      >
>      > I'm now running a machine with head (commit
>      > b32d49cfbaa0437d08e65e7cd7c82c5951b1a852 Jan 25th), poudriere
>     installed
>      > in it, machine is amd64, with an arm64 jail, 14.0-RELEASE, installed
>      > from official distribution binaries (https download method), with
>     cross
>      > tools.
>      >
>      > To make sure everything is aligned I rebuild everything: updated
>     head,
>      > rebuild cross tools in the jail, recompiled all ports for the host
>      > architecture and force reinstalled them, especially
>     qemu-user-static,
>      > cleaned up all packages for the arm64 jail.
>      >
>      > If I missed something important please point it out.
>      >
>      > I have made some more tests and I'm getting python failures in
>     poudriere
>      > like the one described below from time to time (don't have hard
>     stats
>      > but feels like 50% chance). If I get past that it usually is able to
>      > build all the not many packages, but locks up at:
>      >
>      > Creating repository in /tmp/packages:   0%
>      >
> 
>     BTW, forgot to mention last time this worked without issue was around
>     20th December.
> 
> 
> I think this is a bsd-user issue. There is a race somewhere in that code 
> that causes the hangs. I'd love a reproducible test case that is 
> somewhat smaller than python... there are bigger races with the newer 
> stuff and I've not had the time to chase it there either. 😞

First of all thanks for your feedback. It encourages me having someone 
else with better knowledge about this confirm that a race condition is 
actually a possible cause!

Strange this has not been happening up to mid December.

My main and fully reproducible use case is actually mostly with pkg.

at the end of the run poudriere runs `pkg repo` to create the meta files 
and sign the repo. It forks itself (ncpus + 2 I guess, even forcing it 
to 1 worker I see three processes), and then locks up, with all the 
processes stopping using CPU (ps output is in my message)

I guess this can be reproduced with any poudriere repo with at least 
more than ncpus packages in it. can also be reproduced using `poudriere 
pkgclean -u <etc>`

If that does not work I'm not sure how to reproduce it in other ways, 
but I can try  writing some code mocking what pkg seems to be doing, not 
an expert at such things, though.

-- 
Guido Falsi <mad@madpilot.net>