FreeBSD-main-amd64-test - Build #19960 - Still Failing

FreeBSD-main-amd64-test - Build #19960 (d0633af7658fe02bc1af8639ef6dbb0d029c1b65) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19960/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19960/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19960/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

83a103ec429a6dd862a73857ebeeff863a41a34d by tuexen:
libc sctp: improve conformance of sctp_getpaddrs()

d91559564dc647846479824340bdf94b7cf98c05 by vmaffione:
e1000: remove unused ifp backpointer

d0633af7658fe02bc1af8639ef6dbb0d029c1b65 by vmaffione:
em: skip rxcsum offload processing when disabled



The end of the build log:

[...truncated 4.21 MB...]
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x2212a88, rbp = 0x2212aa0 ---

Tracing command zfskern pid 35639 tid 100284 td 0xfffffe009db853a0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063becdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe0063becde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bece20
_sleep() at _sleep+0x2e7/frame 0xfffffe0063becec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0063becef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063becf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063becf30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x2e18a88, rbp = 0x2e18aa0 ---

Tracing command zfskern pid 35639 tid 100285 td 0xfffffe00a6a55000
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9f5dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9f5de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9f5e20
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9f5ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d9f5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9f5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9f5f30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x2a16a88, rbp = 0x2a16aa0 ---

Tracing command zfskern pid 35639 tid 100286 td 0xfffffe009dbe4740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5de9dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5de9de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5de9e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5de9ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a5de9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5de9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5de9f30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x403a88, rbp = 0x403aa0 ---

Tracing command zfskern pid 35639 tid 100287 td 0xfffffe00a6a48ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe009d98cdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d98cde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d98ce20
_sleep() at _sleep+0x2e7/frame 0xfffffe009d98cec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d98cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d98cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d98cf30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x381da88, rbp = 0x381daa0 ---

Tracing command zfskern pid 35639 tid 100288 td 0xfffffe00a6a5ee40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dd5dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dd5de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dd5e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5dd5ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a5dd5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dd5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dd5f30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x7fffdb555a88, rbp = 0x7fffdb555aa0 ---

Tracing command zfskern pid 35639 tid 100289 td 0xfffffe009dbecc80
sched_switch() at sched_switch+0x559/frame 0xfffffe009d99bc20
mi_switch() at mi_switch+0x157/frame 0xfffffe009d99bc40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d99bc80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe009d99bcc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe009d99bd40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe009d99bef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d99bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d99bf30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x4222a88, rbp = 0x4222aa0 ---

Tracing command zfskern pid 35639 tid 100290 td 0xfffffe009dbe3900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5ddfdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5ddfde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5ddfe20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5ddfec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a5ddfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5ddff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5ddff30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x3a1ea88, rbp = 0x3a1eaa0 ---

Tracing command zfskern pid 35639 tid 100291 td 0xfffffe00a6a57ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5db2cc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5db2ce0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5db2d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a5db2d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a5db2d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00a5db2de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00a5db2e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00a5db2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5db2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5db2f30
--- trap 0xc, rip = 0x12a173f5609a, rsp = 0x5029a88, rbp = 0x5029aa0 ---

Tracing command ng_queue pid 24066 tid 101415 td 0xfffffe00a7b23000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7e4ad80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7e4ada0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7e4ade0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a7e4ae80
ngthread() at ngthread+0x88/frame 0xfffffe00a7e4aef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7e4af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7e4af30
--- trap 0xc, rip = 0x307be284609a, rsp = 0x7fffff63fec8, rbp = 0x7fffff63ffc0 ---

Tracing command ng_queue pid 24066 tid 102145 td 0xfffffe00a7b36c80
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c63d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c63da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c63de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c63e80
ngthread() at ngthread+0x88/frame 0xfffffe00a6c63ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c63f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c63f30
--- trap 0xc, rip = 0x36403318e09a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command aiod15 pid 73422 tid 100109 td 0xfffffe009db87020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5d85d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5d85d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5d85dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5d85e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5d85ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5d85ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5d85f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5d85f30
--- trap 0xc, rip = 0x115944f5509a, rsp = 0x7fffff0ac2b8, rbp = 0x7fffff0ac2d0 ---

Tracing command aiod21 pid 73428 tid 100443 td 0xfffffe00a6a603a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cc7d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cc7d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6cc7dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6cc7e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6cc7ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a6cc7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cc7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cc7f30
--- trap 0xc, rip = 0x1d9ea625d09a, rsp = 0x7fffff8caac8, rbp = 0x7fffff8caae0 ---

Tracing command aiod29 pid 73436 tid 100805 td 0xfffffe00a7b35000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6d03d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6d03d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6d03dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6d03e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6d03ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a6d03ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6d03f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6d03f30
--- trap 0xc, rip = 0x2cbd101d509a, rsp = 0x7fffff7e28b8, rbp = 0x7fffff7e28d0 ---

Tracing command aiod32 pid 73439 tid 100118 td 0xfffffe009dbe0e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00765dad60
mi_switch() at mi_switch+0x157/frame 0xfffffe00765dad80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00765dadc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00765dae00
_sleep() at _sleep+0x2ce/frame 0xfffffe00765daea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00765daef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00765daf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00765daf30
--- trap 0xc, rip = 0x10d199e8709a, rsp = 0x7ffffff1c9e8, rbp = 0x7ffffff1cae0 ---

Tracing command soaiod1 pid 73446 tid 101390 td 0xfffffe00a7b2a000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7eccd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7eccd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7eccdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a7ecce10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a7ecceb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a7eccef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7eccf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7eccf30
--- trap 0xc, rip = 0x2cb7c400209a, rsp = 0x7ffffffe3448, rbp = 0x7ffffffe3460 ---

Tracing command soaiod2 pid 73447 tid 101389 td 0xfffffe00a7403900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7f7bd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7f7bd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7f7bdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a7f7be10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a7f7beb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a7f7bef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7f7bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7f7bf30
--- trap 0xc, rip = 0x288e302f809a, rsp = 0x7ffffffeeb68, rbp = 0x7ffffffeeb80 ---

Tracing command soaiod3 pid 73448 tid 101388 td 0xfffffe009db84560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6d80d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6d80d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6d80dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6d80e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a6d80eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a6d80ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6d80f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6d80f30
--- trap 0xc, rip = 0x23d0ea68b09a, rsp = 0x7fffff980a08, rbp = 0x7fffff980a20 ---

Tracing command soaiod4 pid 73449 tid 100103 td 0xfffffe00767dc740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5da3d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5da3d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5da3dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5da3e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5da3eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5da3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5da3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5da3f30
--- trap 0xc, rip = 0x29bd7322709a, rsp = 0x7fffff3da018, rbp = 0x7fffff3da030 ---

Tracing command g_mp_kt pid 16152 tid 102105 td 0xfffffe00a6a47c80
sched_switch() at sched_switch+0x559/frame 0xfffffe00a7f35da0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a7f35dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a7f35e00
_sleep() at _sleep+0x2e7/frame 0xfffffe00a7f35ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe00a7f35ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7f35f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7f35f30
--- trap 0xc, rip = 0x3287d421f09a, rsp = 0x7fffffd37008, rbp = 0x7fffffd37020 ---
db:0:kdb.enter.panic> dump
Dumping 492 out of 4060 MB:..4%..14%..23%..33%..43%..53%..62%..72%..82%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19960' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./run-kyua.sh
x ./disable-zfs-tests.sh
x ./run.sh
x ./disable-notyet-tests.sh
x ./auto-shutdown
x ./disable-dtrace-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins2459531219307073589.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'd0633af7658fe02bc1af8639ef6dbb0d029c1b65', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../d0633af7658fe02bc1af8639ef6dbb0d029c1b65/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address