Re: FreeBSD-main-amd64-test - Build #25976 - Still unstable

On Wed, Jan 29, 2025 at 05:16:50PM +0200, Konstantin Belousov wrote:
> On Wed, Jan 29, 2025 at 05:29:35AM +0000, jenkins-admin@FreeBSD.org wrote:
> > FreeBSD-main-amd64-test - Build #25976 (4da070ce6c015a994ec4ecf3d31ee94810ea19f1) - Still unstable
> > 
> > Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/25976/
> > Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/25976/changes
> > Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/25976/console
> > 
> > Status explanation:
> > "Unstable" - some tests are suspected being broken by the following changes
> > "Still Unstable" - the failing test cases have not been fixed by these
> >                    following changes and this is a notification to note that
> >                    these changes have not been fully tested by the CI system
> > 
> > Change summaries:
> > (Those commits are likely but not certainly responsible)
> > 
> > 6ee34bca48a9e0867d46b24a78855e225d46ddda by kib:
> > crtbegin: accurately check for the end of .dtors
> > 
> > 21502f9a926c7e0c24ce230bb029fde4bf570a14 by kib:
> > crtend: accurately check for the start of .ctors
> > 
> > 51688136b161089b317edc34c3a3e9b40d54a18a by markj:
> > libdtrace: Use designators to initialize the opcode array
> > 
> > 70c067062217a5295ed321427b91cfd857c18b77 by markj:
> > dtrace/arm64: Simplify dtrace_getarg() slightly
> > 
> > 4da070ce6c015a994ec4ecf3d31ee94810ea19f1 by markj:
> > dtrace/arm64: Fix enumeration of FBT return probes
> > 
> > 
> > 
> > The failed test cases:
> > 
> > 4 tests failed.
> > FAILED:  lib.csu.dynamiclib.cxx_constructors.__test_cases_list__
> > 
> > Error Message:
> > Test program did not exit cleanly
> > 
> > FAILED:  lib.csu.dynamiclib.fini_test.__test_cases_list__
> > 
> > Error Message:
> > Test program did not exit cleanly
> > 
> > FAILED:  lib.csu.dynamiclib.init_test.__test_cases_list__
> > 
> > Error Message:
> > Test program did not exit cleanly
> 
> Could anybody explain what does break there?
> I run the 'kyua test -k /usr/tests/lib/csu/Kyuafile' command locally after
> my recent changes to lib/csu/crtbegin/end.c, and it succeeds.
> 
> I believe I saw the sporadic mails with the similar failures before.

The logs from the test suite run are useless, and I cannot reproduce the
problem locally either.

I just remembered that one can fetch the test VM images from
artifact.ci.freebsd.org.  So, I tried that.  The problem is that
/usr/tests/lib/csu/dynamiclib/libh_csu.so is missing, and the test
programs under /usr/tests/lib/csu/dynamiclib link against it, so they
refuse to start.

It looks like the CI scripts build the VM image using artifacts from
"make packagesystem", which I'm unfamiliar with.  It looks like the
tests.txz snapshots are also missing libh_csu.so, so this is some bug in
the release scripts and/or the csu test makefiles.

The failures apparently started with commit
41adc5f29ba6955e09f46e4d2c25da6fcd263ba2, so maybe Ed or Brooks have
some idea where the problem is.