FreeBSD-main-amd64-test - Build #19782 - Still Failing

FreeBSD-main-amd64-test - Build #19782 (522a2aa7613aa43f4d3bb5bdd1d87f966304be34) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19782/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19782/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19782/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

522a2aa7613aa43f4d3bb5bdd1d87f966304be34 by jhb:
Drop "All rights reserved" from a Netflix copyright.



The end of the build log:

[...truncated 4.17 MB...]
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6c45c20
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6c45c40
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6c45c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6c45cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe00a6c45d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00a6c45ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c45f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c45f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffdf9faa88, rbp = 0x7fffdf9faaa0 ---

Tracing command zfskern pid 21244 tid 100356 td 0xfffffe009db951e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00773dcdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00773dcde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00773dce20
_sleep() at _sleep+0x29e/frame 0xfffffe00773dcec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00773dcef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773dcf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773dcf30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffddfeda88, rbp = 0x7fffddfedaa0 ---

Tracing command zfskern pid 21244 tid 100357 td 0xfffffe00a6a7f3a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00773ffcc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00773ffce0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00773ffd20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00773ffd70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00773ffd80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00773ffde0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00773ffe00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00773ffef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773fff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773fff30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd7fbda88, rbp = 0x7fffd7fbdaa0 ---

Tracing command ng_queue pid 51966 tid 101227 td 0xfffffe00a7ba71e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7f94d80
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7f94da0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7f94de0
_sleep() at _sleep+0x29e/frame 0xfffffe00a7f94e80
ngthread() at ngthread+0x88/frame 0xfffffe00a7f94ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7f94f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7f94f30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe868, rbp = 0x7fffffffe960 ---

Tracing command ng_queue pid 51966 tid 102216 td 0xfffffe00a7c18c80
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7ed1d80
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7ed1da0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7ed1de0
_sleep() at _sleep+0x29e/frame 0xfffffe00a7ed1e80
ngthread() at ngthread+0x88/frame 0xfffffe00a7ed1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7ed1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7ed1f30
--- trap 0x16, rip = 0x801415bba, rsp = 0x7fffdf9faa88, rbp = 0x7fffdf9faaa0 ---

Tracing command md0 pid 54050 tid 100115 td 0xfffffe009db943a0
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bd1da0
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bd1dc0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bd1e00
_sleep() at _sleep+0x29e/frame 0xfffffe0063bd1ea0
md_kthread() at md_kthread+0xd5/frame 0xfffffe0063bd1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bd1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bd1f30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe838, rbp = 0x7fffffffe930 ---

Tracing command g_mp_kt pid 44297 tid 101302 td 0xfffffe00a7bb5720
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6c68da0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6c68dc0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6c68e00
_sleep() at _sleep+0x29e/frame 0xfffffe00a6c68ea0
g_multipath_kt() at g_multipath_kt+0x265/frame 0xfffffe00a6c68ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c68f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c68f30
--- trap 0xc, rip = 0x8011a8bba, rsp = 0x7fffffffeae8, rbp = 0x7fffffffeb00 ---

Tracing command sh pid 54829 tid 102541 td 0xfffffe00a761b1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00ac291a10
mi_switch() at mi_switch+0x155/frame 0xfffffe00ac291a30
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00ac291a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00ac291ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00ac291ad0
_sleep() at _sleep+0x294/frame 0xfffffe00ac291b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00ac291c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00ac291e00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe00ac291f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ac291f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120a15a, rsp = 0x7fffffffdae8, rbp = 0x7fffffffdb90 ---

Tracing command aiod13 pid 1589 tid 101310 td 0xfffffe00a7bc2020
sched_switch() at sched_switch+0x578/frame 0xfffffe009d979d60
mi_switch() at mi_switch+0x155/frame 0xfffffe009d979d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe009d979dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe009d979e00
_sleep() at _sleep+0x282/frame 0xfffffe009d979ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe009d979ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d979f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d979f30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffe9d0 ---

Tracing command aiod19 pid 1595 tid 100096 td 0xfffffe00775a6020
sched_switch() at sched_switch+0x578/frame 0xfffffe0063b43d60
mi_switch() at mi_switch+0x155/frame 0xfffffe0063b43d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063b43dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b43e00
_sleep() at _sleep+0x282/frame 0xfffffe0063b43ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b43ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b43f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b43f30
--- trap 0xc, rip = 0x8011a7bba, rsp = 0x7fffffffea08, rbp = 0x7fffffffea20 ---

Tracing command aiod23 pid 1599 tid 101258 td 0xfffffe00a7750560
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6c86d60
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6c86d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6c86dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6c86e00
_sleep() at _sleep+0x282/frame 0xfffffe00a6c86ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a6c86ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c86f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c86f30
--- trap 0xc, rip = 0x8011a7bba, rsp = 0x7fffffffea08, rbp = 0x7fffffffea20 ---

Tracing command aiod30 pid 1606 tid 101334 td 0xfffffe00a7bdc020
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7fe9d60
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7fe9d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7fe9dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a7fe9e00
_sleep() at _sleep+0x282/frame 0xfffffe00a7fe9ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a7fe9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7fe9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7fe9f30
--- trap 0xc, rip = 0x8011a7bba, rsp = 0x7fffffffea08, rbp = 0x7fffffffea20 ---

Tracing command soaiod1 pid 1615 tid 101237 td 0xfffffe00775a43a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6c7cd70
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6c7cd90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6c7cdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6c7ce10
_sleep() at _sleep+0x282/frame 0xfffffe00a6c7ceb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00a6c7cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c7cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c7cf30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command soaiod2 pid 1616 tid 100114 td 0xfffffe009db94ac0
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bd6d70
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bd6d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bd6dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063bd6e10
_sleep() at _sleep+0x282/frame 0xfffffe0063bd6eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe0063bd6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bd6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bd6f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command soaiod3 pid 1617 tid 101232 td 0xfffffe00a7bbee40
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7f99d70
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7f99d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7f99dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a7f99e10
_sleep() at _sleep+0x282/frame 0xfffffe00a7f99eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00a7f99ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7f99f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7f99f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd538, rbp = 0x7fffffffd550 ---

Tracing command soaiod4 pid 1618 tid 100803 td 0xfffffe00a7c31000
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7f53d70
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7f53d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7f53dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a7f53e10
_sleep() at _sleep+0x282/frame 0xfffffe00a7f53eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00a7f53ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7f53f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7f53f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command atf-check pid 54879 tid 101241 td 0xfffffe006a440c80
sched_switch() at sched_switch+0x578/frame 0xfffffe00a5dd9a10
mi_switch() at mi_switch+0x155/frame 0xfffffe00a5dd9a30
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a5dd9a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a5dd9ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a5dd9ad0
_sleep() at _sleep+0x294/frame 0xfffffe00a5dd9b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00a5dd9c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00a5dd9e00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe00a5dd9f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00a5dd9f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80134215a, rsp = 0x7fffffffe648, rbp = 0x7fffffffe680 ---

Tracing command python3.8 pid 54882 tid 100109 td 0xfffffe00a6942ac0 (CPU 0)
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00a5d9b990
vpanic() at vpanic+0x1b8/frame 0xfffffe00a5d9b9f0
panic() at panic+0x43/frame 0xfffffe00a5d9ba50
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00a5d9bab0
trap_pfault() at trap_pfault+0x99/frame 0xfffffe00a5d9bb10
trap() at trap+0x2a7/frame 0xfffffe00a5d9bc20
calltrap() at calltrap+0x8/frame 0xfffffe00a5d9bc20
--- trap 0xc, rip = 0xffffffff80e1d9c7, rsp = 0xfffffe00a5d9bcf0, rbp = 0xfffffe00a5d9bd20 ---
tcp_newtcpcb() at tcp_newtcpcb+0x127/frame 0xfffffe00a5d9bd20
tcp_usr_attach() at tcp_usr_attach+0x103/frame 0xfffffe00a5d9bd50
socreate() at socreate+0x1cf/frame 0xfffffe00a5d9bda0
kern_socket() at kern_socket+0xb8/frame 0xfffffe00a5d9be00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe00a5d9bf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00a5d9bf30
--- syscall (97, FreeBSD ELF64, sys_socket), rip = 0x80076a47a, rsp = 0x7fffffff5ba8, rbp = 0x7fffffff5c80 ---
db:0:kdb.enter.panic> dump
Dumping 435 out of 4058 MB:..4%..12%..23%..34%..41%..52%..63%..74%..81%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19782' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-dtrace-tests.sh
x ./disable-zfs-tests.sh
x ./run.sh
x ./auto-shutdown
x ./disable-notyet-tests.sh
x ./run-kyua.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins2458817806160406433.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '522a2aa7613aa43f4d3bb5bdd1d87f966304be34', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../522a2aa7613aa43f4d3bb5bdd1d87f966304be34/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address