FreeBSD-main-amd64-test - Build #19953 - Still Failing

FreeBSD-main-amd64-test - Build #19953 (a01b8859cbde9f2ded7b58a024d8fb46ada596bb) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19953/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19953/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19953/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

a01b8859cbde9f2ded7b58a024d8fb46ada596bb by tuexen:
sctp: cleanup, no functional change intended.



The end of the build log:

[...truncated 4.26 MB...]
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d9d7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9d7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9d7f30
--- trap 0xc, rip = 0xb05d4b5006a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command zfskern pid 23297 tid 100532 td 0xfffffe00a6a1ae40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dc8dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dc8de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dc8e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5dc8ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a5dc8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dc8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dc8f30
--- trap 0x16, rip = 0x1f88db78a06a, rsp = 0x7fffdf395f48, rbp = 0x7fffdf395f60 ---

Tracing command zfskern pid 23297 tid 100533 td 0xfffffe00a6a2d740
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9c3dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9c3de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9c3e20
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9c3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d9c3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9c3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9c3f30
--- trap 0xc, rip = 0x1f88db78a06a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command zfskern pid 23297 tid 100534 td 0xfffffe00a6a1e740
sched_switch() at sched_switch+0x559/frame 0xfffffe00765fadc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00765fade0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765fae20
_sleep() at _sleep+0x2e7/frame 0xfffffe00765faec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00765faef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00765faf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00765faf30
--- trap 0xc, rip = 0x38e7d17de06a, rsp = 0x13e9f18, rbp = 0x13e9f30 ---

Tracing command zfskern pid 23297 tid 100535 td 0xfffffe00a6a29000
sched_switch() at sched_switch+0x559/frame 0xfffffe009d97dc20
mi_switch() at mi_switch+0x157/frame 0xfffffe009d97dc40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d97dc80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe009d97dcc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe009d97dd40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe009d97def0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d97df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d97df30
--- trap 0xc, rip = 0x38e7d17de06a, rsp = 0x202e38, rbp = 0x202e50 ---

Tracing command zfskern pid 23297 tid 100536 td 0xfffffe00a6a1e020
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9c8dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9c8de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9c8e20
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9c8ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d9c8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9c8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9c8f30
--- trap 0xc, rip = 0x279a63a7e34a, rsp = 0x604f18, rbp = 0x604f40 ---

Tracing command zfskern pid 23297 tid 100537 td 0xfffffe00a6a11000
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9b4cc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9b4ce0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9b4d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe009d9b4d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe009d9b4d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe009d9b4de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe009d9b4e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe009d9b4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9b4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9b4f30
--- trap 0xc, rip = 0x279a63a7e34a, rsp = 0xa06f18, rbp = 0xa06f40 ---

Tracing command ng_queue pid 40411 tid 100904 td 0xfffffe00a6a7e020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f7ed80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f7eda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f7ede0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6f7ee80
ngthread() at ngthread+0x88/frame 0xfffffe00a6f7eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f7ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f7ef30
--- trap 0xc, rip = 0x390cc6b8906a, rsp = 0x7fffff331d88, rbp = 0x7fffff331da0 ---

Tracing command ng_queue pid 40411 tid 101826 td 0xfffffe00a6ad5740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6db2d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6db2da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6db2de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6db2e80
ngthread() at ngthread+0x88/frame 0xfffffe00a6db2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6db2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6db2f30
--- trap 0xc, rip = 0xc29bb44406a, rsp = 0xcc67f48, rbp = 0xcc67f60 ---

Tracing command aiod9 pid 89760 tid 100093 td 0xfffffe00767b8720
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b77d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b77d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b77dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b77e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b77ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b77ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b77f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b77f30
--- trap 0xc, rip = 0xab2f411706a, rsp = 0x7fffff6cf848, rbp = 0x7fffff6cf860 ---

Tracing command aiod14 pid 89765 tid 100857 td 0xfffffe00a6a65c80
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9a5d60
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9a5d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9a5dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe009d9a5e00
_sleep() at _sleep+0x2ce/frame 0xfffffe009d9a5ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe009d9a5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9a5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9a5f30
--- trap 0xc, rip = 0x21028f6e406a, rsp = 0x7fffff6daaa8, rbp = 0x7fffff6daac0 ---

Tracing command aiod31 pid 89782 tid 100810 td 0xfffffe00a6a1d900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5daad60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5daad80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5daadc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5daae00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5daaea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5daaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5daaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5daaf30
--- trap 0xc, rip = 0x1d60d9c4a64a, rsp = 0x7fffff2a2608, rbp = 0x7fffff2a2ab0 ---

Tracing command aiod32 pid 89783 tid 100847 td 0xfffffe00a6a2b3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f47d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f47d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f47dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6f47e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6f47ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a6f47ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f47f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f47f30
--- trap 0xc, rip = 0x33d55997b06a, rsp = 0x7fffffa959a8, rbp = 0x7fffffa959c0 ---

Tracing command soaiod1 pid 89790 tid 100851 td 0xfffffe00a6a51900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f01d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f01d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f01dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6f01e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6f01eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a6f01ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f01f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f01f30
--- trap 0xc, rip = 0x29cde0cf206a, rsp = 0x7fffff727dc8, rbp = 0x7fffff727de0 ---

Tracing command soaiod2 pid 89791 tid 100846 td 0xfffffe00a6a64e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f4cd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f4cd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f4cdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6f4ce10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6f4ceb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a6f4cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f4cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f4cf30
--- trap 0xc, rip = 0x3bab6082306a, rsp = 0x7fffff3b3358, rbp = 0x7fffff3b3370 ---

Tracing command soaiod3 pid 89792 tid 100924 td 0xfffffe00a6aa3720
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6fbfd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6fbfd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6fbfdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6fbfe10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6fbfeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a6fbfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6fbff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6fbff30
--- trap 0xc, rip = 0x2da0b26c506a, rsp = 0x7fffff2c32c8, rbp = 0x7fffff2c32e0 ---

Tracing command soaiod4 pid 89793 tid 100930 td 0xfffffe00a6ab03a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cd1d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cd1d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6cd1dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6cd1e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6cd1eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a6cd1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cd1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cd1f30
--- trap 0xc, rip = 0x2d16512de06a, rsp = 0x7fffff6c1eb8, rbp = 0x7fffff6c1ed0 ---

Tracing command g_mp_kt pid 32497 tid 100918 td 0xfffffe00a6aaf560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c77da0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c77dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c77e00
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c77ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe00a6c77ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c77f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c77f30
--- trap 0xc, rip = 0xcdb453a006a, rsp = 0x7fffffed5f68, rbp = 0x7fffffed5f80 ---

Tracing command kyua pid 87957 tid 102221 td 0xfffffe00a7a713a0 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00097cde00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00097cde10
trap() at trap+0x56/frame 0xfffffe00097cdf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00097cdf20
--- trap 0x13, rip = 0xffffffff80c6ef7e, rsp = 0xfffffe00ac7fe6f0, rbp = 0xfffffe00ac7fe720 ---
witness_unlock() at witness_unlock+0xe/frame 0xfffffe00ac7fe720
_rw_wunlock_cookie() at _rw_wunlock_cookie+0x50/frame 0xfffffe00ac7fe740
pmap_remove_pages() at pmap_remove_pages+0x60b/frame 0xfffffe00ac7fe890
exec_new_vmspace() at exec_new_vmspace+0x209/frame 0xfffffe00ac7fe900
exec_elf64_imgact() at exec_elf64_imgact+0x877/frame 0xfffffe00ac7fe9f0
kern_execve() at kern_execve+0x70d/frame 0xfffffe00ac7fed80
sys_execve() at sys_execve+0x5a/frame 0xfffffe00ac7fee00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00ac7fef30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ac7fef30
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x19f6246fa6ea, rsp = 0x7fffffce8eb8, rbp = 0x7fffffce8ec0 ---
db:0:kdb.enter.panic> dump
Dumping 483 out of 4060 MB:..4%..14%..24%..34%..44%..54%..63%..73%..83%..93%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19953' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./run-kyua.sh
x ./run.sh
x ./disable-dtrace-tests.sh
x ./auto-shutdown
x ./disable-zfs-tests.sh
x ./disable-notyet-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins5269443317900727273.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'a01b8859cbde9f2ded7b58a024d8fb46ada596bb', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../a01b8859cbde9f2ded7b58a024d8fb46ada596bb/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address