Strange network/socket anomalies since about a month

Hi,

I see a higher failure rate of socket/network related stuff since a 
while. Those failures are transient. Directly executing the same thing 
again may or may not result in success/failure. I'm not able to 
reproduce this at will. Sometimes they show up.

Examples:
  - poudriere runs with the sccache overlay (like ccache but also works 
for rust) sometimes fail to create the communication socket and as such 
the build fails. I have 3 different poudriere bulk runs after each other 
in my build script, and when the first one fails, the second and third 
still run. If the first fails due to the sccache issue, the second and 
3rd may or may not fail. Sometimes the first fails and the rest is ok. 
Sometimes all fail, and if I then run one by hand it works (the script 
does the same as the manual run, the script is simply a "for type in A B 
C; do; poudriere bulk -O sccache -j $type -f  ${type}.pkglist; done" 
which I execute from the same shell, and the script doesn't do 
env-sanityzing).
  - A webmail interface (inet / local net -> nginx (rev-proxy) -> nginx 
(webmail service) -> php -> imap) sees intermittent issues sometimes. 
Opening the same email directly again afterwards normally works. I've 
also seen transient issues with pgp signing (webmail interface -> gnupg 
/ gpg-agent on the server), simply hitting send again after a failure 
works fine.

Gleb, could this be related to the socket stuff you did 2 weeks ago? My 
world is from 2024-04-17-112537. I do notice this since at least then, 
but I'm not sure if they where there before that and I simply didn't 
notice them. They are surely "new recently", that amount of issues I 
haven's seen in January. The last two updates of current I did before 
the last one where on 2024-03-31-120210 and 2024-04-08-112551.

I could also imagine that some memory related transient failure could 
cause this, but with >3 GB free I do not expect this. Important here may 
be that I have https://reviews.freebsd.org/D40575 in my tree, which is 
memory related, but it's only a metric to quantify memory fragmentation.

Any ideas how to track this down more easily than running the entire 
poudriere in ktrace (e.g. a hint/script which dtrace probes to use)?

Bye,
Alexander.

-- 
http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF