FreeBSD-main-amd64-test - Build #19849 - Still Failing

FreeBSD-main-amd64-test - Build #19849 (b9569ba05e20cdcf252963a0ce6d40aa2fa97848) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19849/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19849/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19849/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

b9569ba05e20cdcf252963a0ce6d40aa2fa97848 by grehan:
Remove myself from bhyve maintenance; ENOTIME.



The end of the build log:

[...truncated 3.92 MB...]

Tracing command ng_queue pid 51877 tid 100865 td 0xfffffe00a6a2cc80
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c0ed80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c0eda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c0ede0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c0ee80
ngthread() at ngthread+0x88/frame 0xfffffe00a6c0eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c0ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c0ef30
--- trap 0x16, rip = 0x2d4a08b59d9a, rsp = 0x604a88, rbp = 0x604aa0 ---

Tracing command sh pid 725 tid 100090 td 0xfffffe00775a8c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b52a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b52a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b52a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b52ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b52ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b52b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b52c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b52e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b52f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b52f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x17926f61033a, rsp = 0x7fffffc73bf8, rbp = 0x7fffffc73cb0 ---

Tracing command sh pid 738 tid 100105 td 0xfffffe00775b8740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dcaa10
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dcaa30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dcaa70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a5dcaac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a5dcaad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00a5dcab70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00a5dcac00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00a5dcae00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00a5dcaf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00a5dcaf30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x280568e4633a, rsp = 0x7fffffe614a8, rbp = 0x7fffffe61560 ---

Tracing command sh pid 745 tid 100100 td 0xfffffe0009988900
sched_switch() at sched_switch+0x559/frame 0xfffffe0063a40a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063a40a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063a40a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063a40ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063a40ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063a40b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063a40c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063a40e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063a40f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063a40f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x21acc77ea33a, rsp = 0x7fffff3212f8, rbp = 0x7fffff3213b0 ---

Tracing command kyua pid 746 tid 100088 td 0xfffffe00775a9ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b5ca10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b5ca30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b5ca70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b5cac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b5cad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b5cb70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b5cc00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b5ce00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b5cf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b5cf30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1af7ff0fb33a, rsp = 0x7fffffcf3738, rbp = 0x7fffffcf3770 ---

Tracing command atf-sh pid 99245 tid 101560 td 0xfffffe00ab648c80 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00097cbe00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00097cbe10
trap() at trap+0x56/frame 0xfffffe00097cbf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00097cbf20
--- trap 0x13, rip = 0xffffffff80f75bc0, rsp = 0xfffffe00ab55ec58, rbp = 0xfffffe00ab55ece0 ---
vm_map_find_min() at vm_map_find_min/frame 0xfffffe00ab55ece0
kern_mmap() at kern_mmap+0x5c7/frame 0xfffffe00ab55edc0
sys_mmap() at sys_mmap+0x42/frame 0xfffffe00ab55ee00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00ab55ef30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ab55ef30
--- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x311f215be17a, rsp = 0x7fffffba59f8, rbp = 0x7fffffba5a20 ---

Tracing command g_mp_kt pid 44023 tid 100134 td 0xfffffe009dbb4e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bc3da0
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bc3dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bc3e00
_sleep() at _sleep+0x2e7/frame 0xfffffe0063bc3ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe0063bc3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bc3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bc3f30
--- trap 0xc, rip = 0x26f641f3ad9a, rsp = 0x7fffffc6d6a8, rbp = 0x7fffffc6d7a0 ---

Tracing command aiod1 pid 1303 tid 100089 td 0xfffffe00775a93a0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b57d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b57d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b57dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b57e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b57ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b57ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b57f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b57f30
--- trap 0xc, rip = 0x197acdd9d9a, rsp = 0x7fffff41b128, rbp = 0x7fffff41b140 ---

Tracing command aiod8 pid 1310 tid 100103 td 0xfffffe009db5d000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dcfd60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dcfd80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dcfdc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5dcfe00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5dcfea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5dcfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dcff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dcff30
--- trap 0xc, rip = 0x3732d5861d9a, rsp = 0x7fffffff25b8, rbp = 0x7fffffff25d0 ---

Tracing command aiod22 pid 1324 tid 100124 td 0xfffffe00775b63a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5db1d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5db1d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5db1dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5db1e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5db1ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5db1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5db1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5db1f30
--- trap 0, rip = 0, rsp = 0, rbp = 0x22996ded561 ---

Tracing command aiod27 pid 1329 tid 100129 td 0xfffffe009dbb8740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5d98d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5d98d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5d98dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5d98e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5d98ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5d98ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5d98f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5d98f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command soaiod1 pid 1341 tid 100112 td 0xfffffe009db60900
sched_switch() at sched_switch+0x559/frame 0xfffffe00773edd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00773edd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773eddd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00773ede10
_sleep() at _sleep+0x2ce/frame 0xfffffe00773edeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00773edef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773edf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773edf30
--- trap 0xc, rip = 0x3ad37c934d9a, rsp = 0x7fffff9bdc28, rbp = 0x7fffff9bdc40 ---

Tracing command soaiod2 pid 1342 tid 100094 td 0xfffffe00775a7000
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b3ed70
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b3ed90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b3edd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b3ee10
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b3eeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe0063b3eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b3ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b3ef30
--- trap 0xc, rip = 0x1e46c2429d9a, rsp = 0x7fffffa16ec8, rbp = 0x7fffffa16ee0 ---

Tracing command soaiod3 pid 1343 tid 100111 td 0xfffffe009db61020
sched_switch() at sched_switch+0x559/frame 0xfffffe00773f2d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00773f2d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773f2dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00773f2e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00773f2eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00773f2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773f2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773f2f30
--- trap 0xc, rip = 0x1bdff0facd9a, rsp = 0x7fffff103f88, rbp = 0x7fffff103fa0 ---

Tracing command soaiod4 pid 1344 tid 100109 td 0xfffffe00775ab020
sched_switch() at sched_switch+0x559/frame 0xfffffe00773fcd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00773fcd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773fcdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00773fce10
_sleep() at _sleep+0x2ce/frame 0xfffffe00773fceb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00773fcef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773fcf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773fcf30
--- trap 0xc, rip = 0x288df33a1d9a, rsp = 0x7fffff73ebf8, rbp = 0x7fffff73ec10 ---

Tracing command KTLS pid 46987 tid 100824 td 0xfffffe00a6a2d3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c13d00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c13d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c13d60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c13e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a6c13ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c13f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c13f30
--- trap 0xc, rip = 0x22beecb95d9a, rsp = 0x7fffff497b78, rbp = 0x7fffff497c30 ---

Tracing command KTLS pid 46987 tid 100833 td 0xfffffe009dbe7740
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9fbd00
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9fbd20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9fbd60
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9fbe00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe009d9fbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9fbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9fbf30
--- trap 0xc, rip = 0x3cf34e75c19a, rsp = 0x102ea8, rbp = 0x102ef0 ---

Tracing command KTLS pid 46987 tid 100834 td 0xfffffe009dbe61e0
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9ecd40
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9ecd60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9ecda0
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9ece40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe009d9ecef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9ecf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9ecf30
--- trap 0x16, rip = 0x318bb7712d9a, rsp = 0x403a88, rbp = 0x403aa0 ---
db:0:kdb.enter.panic> dump
Dumping 461 out of 4058 MB:..4%..11%..21%..32%..42%..53%..63%..73%..84%..91%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19849' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-zfs-tests.sh
x ./disable-dtrace-tests.sh
x ./auto-shutdown
x ./run.sh
x ./run-kyua.sh
x ./disable-notyet-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins3618957081255163839.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'b9569ba05e20cdcf252963a0ce6d40aa2fa97848', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../b9569ba05e20cdcf252963a0ce6d40aa2fa97848/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address