FreeBSD-main-amd64-test - Build #19969 - Failure

FreeBSD-main-amd64-test - Build #19969 (8f82dc8dd3ffad550725f87ae69c0ad10fedf443) - Failure

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19969/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19969/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19969/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

8f82dc8dd3ffad550725f87ae69c0ad10fedf443 by bdrewery:
hyperv: Flag hn and storvsc statistics with CTLFLAG_STATS.



The end of the build log:

[...truncated 3.92 MB...]
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5df8ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5df8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5df8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5df8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command soaiod1 pid 2203 tid 100136 td 0xfffffe009dbd91e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5ddad70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5ddad90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5ddadd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5ddae10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5ddaeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5ddaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5ddaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5ddaf30
--- trap 0xc, rip = 0x215300f5e09a, rsp = 0x7fffff87de08, rbp = 0x7fffff87de20 ---

Tracing command soaiod2 pid 2204 tid 100135 td 0xfffffe009dbd9900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5ddfd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5ddfd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5ddfdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5ddfe10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5ddfeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5ddfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5ddff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5ddff30
--- trap 0xc, rip = 0x35e9677d809a, rsp = 0x7fffff43e118, rbp = 0x7fffff43e130 ---

Tracing command soaiod3 pid 2205 tid 100134 td 0xfffffe009dbda020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5de4d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5de4d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5de4dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5de4e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5de4eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5de4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5de4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5de4f30
--- trap 0xc, rip = 0x1aec8546e09a, rsp = 0x7ffffff35778, rbp = 0x7ffffff35790 ---

Tracing command soaiod4 pid 2206 tid 100133 td 0xfffffe009dbda740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5de9d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5de9d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5de9dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5de9e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5de9eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5de9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5de9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5de9f30
--- trap 0xc, rip = 0x391855a6209a, rsp = 0x7ffffffbec38, rbp = 0x7ffffffbec50 ---

Tracing command devd pid 473 tid 100105 td 0xfffffe009dba9720
sched_switch() at sched_switch+0x559/frame 0xfffffe00765e4a60
mi_switch() at mi_switch+0x157/frame 0xfffffe00765e4a80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765e4ac0
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00765e4b10
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00765e4b50
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19a/frame 0xfffffe00765e4bd0
kern_select() at kern_select+0xa6f/frame 0xfffffe00765e4dc0
sys_select() at sys_select+0x56/frame 0xfffffe00765e4e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00765e4f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00765e4f30
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x28b1fa, rsp = 0x7fffff17e568, rbp = 0x7fffff180700 ---

Tracing command auditd pid 673 tid 100087 td 0xfffffe00767b61e0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b95b50
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b95b70
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b95bb0
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b95c00
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b95c10
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b95cb0
audit_read() at audit_read+0x9c/frame 0xfffffe0063b95ce0
devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0063b95d40
dofileread() at dofileread+0x81/frame 0xfffffe0063b95d90
sys_read() at sys_read+0xc0/frame 0xfffffe0063b95e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b95f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b95f30
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x9b77b4ad45a, rsp = 0x7fffffe0dcd8, rbp = 0x7fffffe0dd20 ---

Tracing command sh pid 725 tid 100090 td 0xfffffe00767b4c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b86a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b86a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b86a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b86ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b86ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b86b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b86c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b86e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b86f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b86f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x9afb71a763a, rsp = 0x7fffff467d78, rbp = 0x7fffff467e30 ---

Tracing command sh pid 738 tid 100108 td 0xfffffe00767d7020
sched_switch() at sched_switch+0x559/frame 0xfffffe00765d5a10
mi_switch() at mi_switch+0x157/frame 0xfffffe00765d5a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765d5a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00765d5ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00765d5ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00765d5b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00765d5c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00765d5e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00765d5f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00765d5f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1084a84d663a, rsp = 0x7fffff9f4718, rbp = 0x7fffff9f47d0 ---

Tracing command sh pid 745 tid 100111 td 0xfffffe00767d5ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00765c6a10
mi_switch() at mi_switch+0x157/frame 0xfffffe00765c6a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765c6a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00765c6ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00765c6ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00765c6b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00765c6c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00765c6e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00765c6f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00765c6f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1044bff7463a, rsp = 0x7fffff98df28, rbp = 0x7fffff98dfe0 ---

Tracing command kyua pid 746 tid 100112 td 0xfffffe00767d53a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00765c1a10
mi_switch() at mi_switch+0x157/frame 0xfffffe00765c1a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765c1a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00765c1ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00765c1ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00765c1b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00765c1c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00765c1e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00765c1f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00765c1f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x313c1085863a, rsp = 0x7fffffa96798, rbp = 0x7fffffa967d0 ---

Tracing command KTLS pid 47877 tid 100125 td 0xfffffe009dbd6e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063becd00
mi_switch() at mi_switch+0x157/frame 0xfffffe0063becd20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063becd60
_sleep() at _sleep+0x2e7/frame 0xfffffe0063bece00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe0063becef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063becf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063becf30
--- trap 0xc, rip = 0x345bf1d1309a, rsp = 0x7fffffb11368, rbp = 0x7fffffb11380 ---

Tracing command KTLS pid 47877 tid 100850 td 0xfffffe00a6a63720
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c5ed00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c5ed20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c5ed60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c5ee00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a6c5eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c5ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c5ef30
--- trap 0x16, rip = 0x91993ad609a, rsp = 0x604a88, rbp = 0x604aa0 ---

Tracing command KTLS pid 47877 tid 100851 td 0xfffffe00a6a49740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c4fd40
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c4fd60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c4fda0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c4fe40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe00a6c4fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c4ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c4ff30
--- trap 0x16, rip = 0x264212aa909a, rsp = 0x805a88, rbp = 0x805aa0 ---

Tracing command atf-sh pid 1065 tid 100928 td 0xfffffe00a806c740 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00097cde00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00097cde10
trap() at trap+0x56/frame 0xfffffe00097cdf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00097cdf20
--- trap 0x13, rip = 0xffffffff811858b1, rsp = 0xfffffe00a6ce3890, rbp = 0xfffffe00a6ce3930 ---
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x1/frame 0xfffffe00a6ce3930
namei() at namei+0x3b8/frame 0xfffffe00a6ce39f0
kern_execve() at kern_execve+0x268/frame 0xfffffe00a6ce3d80
sys_execve() at sys_execve+0x5a/frame 0xfffffe00a6ce3e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00a6ce3f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00a6ce3f30
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x180a7dd3371a, rsp = 0x7fffffcbce58, rbp = 0x7fffffcbce60 ---

Tracing command ng_queue pid 52807 tid 100138 td 0xfffffe00a6a3e740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dd0d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dd0da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dd0de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5dd0e80
ngthread() at ngthread+0x88/frame 0xfffffe00a5dd0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dd0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dd0f30
--- trap 0xc, rip = 0x80ea9f2209a, rsp = 0x7fffff28a418, rbp = 0x7fffff28a510 ---

Tracing command ng_queue pid 52807 tid 100882 td 0xfffffe00a6a45e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c40d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c40da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c40de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c40e80
ngthread() at ngthread+0x88/frame 0xfffffe00a6c40ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c40f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c40f30
--- trap 0xc, rip = 0x309381da09a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command g_mp_kt pid 44893 tid 100127 td 0xfffffe009dbd6000
sched_switch() at sched_switch+0x559/frame 0xfffffe0063be2da0
mi_switch() at mi_switch+0x157/frame 0xfffffe0063be2dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063be2e00
_sleep() at _sleep+0x2e7/frame 0xfffffe0063be2ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe0063be2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063be2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063be2f30
--- trap 0xc, rip = 0x3bd56b6e509a, rsp = 0x7fffff920c38, rbp = 0x7fffff920c50 ---
db:0:kdb.enter.panic> dump
Dumping 471 out of 4060 MB:..4%..11%..21%..31%..41%..51%..62%..72%..82%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19969' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-dtrace-tests.sh
x ./disable-notyet-tests.sh
x ./disable-zfs-tests.sh
x ./auto-shutdown
x ./run-kyua.sh
x ./run.sh
+ rm -f test-report.txt test-report.xml
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins3123271209503729160.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '8f82dc8dd3ffad550725f87ae69c0ad10fedf443', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../8f82dc8dd3ffad550725f87ae69c0ad10fedf443/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address