FreeBSD-main-amd64-test - Build #19905 - Still Failing

FreeBSD-main-amd64-test - Build #19905 (9c983ad791c3ca8213cae949f9ad381d1309ad8e) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19905/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19905/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19905/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

9c983ad791c3ca8213cae949f9ad381d1309ad8e by emaste:
arch.7: update applicable FreeBSD versions to 12.0 and later



The end of the build log:

[...truncated 3.92 MB...]
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00085fff30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00085fff30
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x1d5b8cf4315a, rsp = 0x7fffffc18b48, rbp = 0x7fffffc18b90 ---

Tracing command sh pid 725 tid 100090 td 0xfffffe00767b9c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b86a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b86a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b86a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b86ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b86ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b86b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b86c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b86e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b86f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b86f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2ae1fd04033a, rsp = 0x7fffff1a3158, rbp = 0x7fffff1a3210 ---

Tracing command ng_queue pid 51937 tid 100109 td 0xfffffe009db6b020
sched_switch() at sched_switch+0x559/frame 0xfffffe00765e6d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00765e6da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765e6de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00765e6e80
ngthread() at ngthread+0x88/frame 0xfffffe00765e6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00765e6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00765e6f30
--- trap 0xc, rip = 0x3d68af578d9a, rsp = 0x7fffff6574e8, rbp = 0x7fffff6575e0 ---

Tracing command ng_queue pid 51937 tid 100865 td 0xfffffe00a6a70900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c36d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c36da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c36de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c36e80
ngthread() at ngthread+0x88/frame 0xfffffe00a6c36ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c36f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c36f30
--- trap 0xa5a5a5a5, rip = 0xa5a5a5a5a5a5a5a5, rsp = 0, rbp = 0x13ccef276b38 ---

Tracing command sh pid 738 tid 100096 td 0xfffffe00767c2560
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bc7a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bc7a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bc7a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063bc7ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063bc7ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063bc7b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063bc7c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063bc7e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063bc7f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063bc7f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0xdafeb76633a, rsp = 0x7fffff480ef8, rbp = 0x7fffff480fb0 ---

Tracing command sh pid 745 tid 100097 td 0xfffffe00767c1e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bc2a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bc2a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bc2a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063bc2ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063bc2ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063bc2b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063bc2c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063bc2e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063bc2f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063bc2f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x30374202533a, rsp = 0x7fffffa18c18, rbp = 0x7fffffa18cd0 ---

Tracing command kyua pid 746 tid 100095 td 0xfffffe00767c2c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bcca10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bcca30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bcca70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063bccac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063bccad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063bccb70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063bccc00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063bcce00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063bccf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063bccf30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x13d4f337d33a, rsp = 0x7fffffd8fdf8, rbp = 0x7fffffd8fe30 ---

Tracing command g_mp_kt pid 44023 tid 100125 td 0xfffffe00767c3ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5d8fda0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5d8fdc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5d8fe00
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5d8fea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe00a5d8fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5d8ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5d8ff30
--- trap 0xc, rip = 0x216f075dbd9a, rsp = 0x7fffff39eed8, rbp = 0x7fffff39efd0 ---

Tracing command aiod14 pid 1316 tid 100120 td 0xfffffe009dbe7560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5deed60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5deed80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5deedc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5deee00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5deeea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5deeef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5deef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5deef30
--- trap 0xc, rip = 0x95899774d9a, rsp = 0x7fffffa57578, rbp = 0x7fffffa57590 ---

Tracing command aiod21 pid 1323 tid 100124 td 0xfffffe00767c41e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5d94d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5d94d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5d94dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5d94e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5d94ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5d94ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5d94f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5d94f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command aiod27 pid 1329 tid 100088 td 0xfffffe00767baac0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b90d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b90d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b90dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b90e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b90ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b90ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b90f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b90f30
--- trap 0xc, rip = 0x3a5143d64d9a, rsp = 0x7fffffd241c8, rbp = 0x7fffffd241e0 ---

Tracing command aiod29 pid 1331 tid 100093 td 0xfffffe00767b8720
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b77d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b77d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b77dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b77e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b77ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b77ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b77f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b77f30
--- trap 0xc, rip = 0x361856413d9a, rsp = 0x7fffff46f258, rbp = 0x7fffff46f270 ---

Tracing command soaiod1 pid 1341 tid 100134 td 0xfffffe00a6a24020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dcbd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dcbd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dcbdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5dcbe10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5dcbeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5dcbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dcbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dcbf30
--- trap 0xc, rip = 0xeedcebe8d9a, rsp = 0x7fffff599428, rbp = 0x7fffff599440 ---

Tracing command soaiod2 pid 1342 tid 100133 td 0xfffffe00a6a24740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dd0d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dd0d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dd0dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5dd0e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5dd0eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5dd0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dd0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dd0f30
--- trap 0xc, rip = 0x1a99473a3d9a, rsp = 0x7fffff3bb3d8, rbp = 0x7fffff3bb3f0 ---

Tracing command soaiod3 pid 1343 tid 100132 td 0xfffffe00a6a51000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dd5d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dd5d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dd5dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5dd5e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5dd5eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5dd5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dd5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dd5f30
--- trap 0xc, rip = 0x88c9579ad9a, rsp = 0x7fffff3faa18, rbp = 0x7fffff3faa30 ---

Tracing command soaiod4 pid 1344 tid 100131 td 0xfffffe009db6b740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5ddad70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5ddad90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5ddadd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5ddae10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5ddaeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5ddaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5ddaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5ddaf30
--- trap 0xc, rip = 0x1f50af565d9a, rsp = 0x7fffff92a478, rbp = 0x7fffff92a490 ---

Tracing command KTLS pid 47007 tid 100135 td 0xfffffe00a6a23900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dc6d00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dc6d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dc6d60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5dc6e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a5dc6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dc6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dc6f30
--- trap 0xc, rip = 0x3adf2ce5ed9a, rsp = 0x7fffff3cf1f8, rbp = 0x7fffff3cf210 ---

Tracing command KTLS pid 47007 tid 100833 td 0xfffffe00a6a20000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c4fd00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c4fd20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c4fd60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c4fe00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a6c4fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c4ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c4ff30
--- trap 0x16, rip = 0x26dc6e2c1d9a, rsp = 0x805a88, rbp = 0x805aa0 ---

Tracing command KTLS pid 47007 tid 100834 td 0xfffffe00a6a643a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c59d40
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c59d60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c59da0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c59e40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe00a6c59ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c59f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c59f30
--- trap 0x16, rip = 0x26dc6e2c1d9a, rsp = 0xa06a88, rbp = 0xa06aa0 ---
db:0:kdb.enter.panic> dump
Dumping 466 out of 4060 MB:..4%..11%..21%..31%..42%..52%..62%..73%..83%..93%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19905' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./auto-shutdown
x ./run-kyua.sh
x ./disable-notyet-tests.sh
x ./run.sh
x ./disable-dtrace-tests.sh
x ./disable-zfs-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins6391716864332348679.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '9c983ad791c3ca8213cae949f9ad381d1309ad8e', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../9c983ad791c3ca8213cae949f9ad381d1309ad8e/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address