FreeBSD-main-amd64-test - Build #19853 - Failure

FreeBSD-main-amd64-test - Build #19853 (ff94500855c16d0d9cc18aa8b0ba73ea94020c56) - Failure

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19853/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19853/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19853/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

322b341d002945fae9151b8beba6f420db61037d by imp:
Remove mips from universe

a721ac948e9bec965a8a0434acede2a672b41a51 by imp:
Document that 13.x is the end of the line for FreeBSD/mips

27a04f59646bea70e8461095eb835e92066702ef by imp:
Add warning that MIPS is being removed in FreeBSD 14.0

fb8588d2cb449518a9e8dc75ef34ed2094c68835 by glebius:
tcp_timewait: use on stack struct tcptw as last resort

ff94500855c16d0d9cc18aa8b0ba73ea94020c56 by glebius:
Add tcp_freecb() - single place to free tcpcb.



The end of the build log:

[...truncated 4.25 MB...]
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6ce0de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6ce0e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6ce0ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6ce0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6ce0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6ce0f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100256 td 0xfffffe00a6ae0900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cdbdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cdbde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6cdbe20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6cdbec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6cdbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cdbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cdbf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100257 td 0xfffffe00a6ae01e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cd6dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cd6de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6cd6e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6cd6ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6cd6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cd6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cd6f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100258 td 0xfffffe00a6adfac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cd1dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cd1de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6cd1e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6cd1ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6cd1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cd1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cd1f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100259 td 0xfffffe00a6bc51e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a760edc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a760ede0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a760ee20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a760eec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a760eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a760ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a760ef30
--- trap 0xa5a5a5a5, rip = 0x20120880f880, rsp = 0, rbp = 0x2009fe4b60c0 ---

Tracing command zfskern pid 48193 tid 100260 td 0xfffffe00a6bc4ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7609dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7609de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7609e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a7609ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a7609ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7609f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7609f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100261 td 0xfffffe00a6bc43a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7604dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7604de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7604e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a7604ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a7604ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7604f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7604f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100262 td 0xfffffe00a6bc3c80
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6dfddc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6dfdde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6dfde20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6dfdec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6dfdef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6dfdf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6dfdf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100263 td 0xfffffe00a6bc3560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6df8dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6df8de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6df8e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6df8ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6df8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6df8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6df8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100264 td 0xfffffe00a6bc2e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6df3dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6df3de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6df3e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6df3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6df3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6df3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6df3f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100265 td 0xfffffe00a6bc2720
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6deedc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6deede0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6deee20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6deeec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6deeef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6deef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6deef30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100266 td 0xfffffe00a6bc2000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6de9dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6de9de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6de9e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6de9ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6de9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6de9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6de9f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100267 td 0xfffffe00a6bcec80
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6de4dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6de4de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6de4e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6de4ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6de4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6de4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6de4f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100268 td 0xfffffe00a6bce560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6ddfc20
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6ddfc40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6ddfc80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6ddfcc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe00a6ddfd40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00a6ddfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6ddff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6ddff30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100269 td 0xfffffe00a6bcde40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6ddadc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6ddade0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6ddae20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6ddaec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6ddaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6ddaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6ddaf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 48193 tid 100270 td 0xfffffe00a6bcd720
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6dd5cc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6dd5ce0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6dd5d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a6dd5d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a6dd5d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00a6dd5de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00a6dd5e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00a6dd5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6dd5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6dd5f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command aiod1 pid 48201 tid 100124 td 0xfffffe009dbdde40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5df7d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5df7d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5df7dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5df7e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5df7ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5df7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5df7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5df7f30
--- trap 0xc, rip = 0x2e804b826d9a, rsp = 0x7fffff9fcf88, rbp = 0x7fffff9fd080 ---

Tracing command aiod4 pid 48204 tid 100092 td 0xfffffe00775a1e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b48d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b48d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b48dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b48e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b48ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b48ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b48f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b48f30
--- trap 0xc, rip = 0x274c8eeecd9a, rsp = 0x7fffff507348, rbp = 0x7fffff507440 ---

Tracing command ng_queue pid 36641 tid 102185 td 0xfffffe00a6bcf3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a9aefd80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a9aefda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a9aefde0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a9aefe80
ngthread() at ngthread+0x88/frame 0xfffffe00a9aefef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a9aeff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a9aeff30
--- trap 0xc, rip = 0x21824adf0d9a, rsp = 0x7fffffdf7378, rbp = 0x7fffffdf7470 ---

Tracing command ng_queue pid 36641 tid 102208 td 0xfffffe00a99231e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a9b3fd80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a9b3fda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a9b3fde0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a9b3fe80
ngthread() at ngthread+0x88/frame 0xfffffe00a9b3fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a9b3ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a9b3ff30
--- trap 0x16, rip = 0x3ce7561d0d9a, rsp = 0x805a88, rbp = 0x805aa0 ---
db:0:kdb.enter.panic> dump
Dumping 487 out of 4058 MB:..4%..14%..23%..33%..43%..53%..63%..73%..82%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19853' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./run.sh
x ./disable-notyet-tests.sh
x ./disable-zfs-tests.sh
x ./run-kyua.sh
x ./auto-shutdown
x ./disable-dtrace-tests.sh
+ rm -f test-report.txt test-report.xml
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins334146739254417366.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'ff94500855c16d0d9cc18aa8b0ba73ea94020c56', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../ff94500855c16d0d9cc18aa8b0ba73ea94020c56/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address