FreeBSD-main-amd64-test - Build #20420 - Still Failing

FreeBSD-main-amd64-test - Build #20420 (89d57b94d776877f77cc04752e449dac57a14618) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/20420/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/20420/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/20420/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

89d57b94d776877f77cc04752e449dac57a14618 by asomers:
fusefs: implement VOP_DEALLOCATE



The end of the build log:

[...truncated 4.31 MB...]
sched_switch() at sched_switch+0x559/frame 0xfffffe009719cdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009719cde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009719ce20
_sleep() at _sleep+0x2e7/frame 0xfffffe009719cec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009719cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009719cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009719cf30
--- trap 0x16, rip = 0x1104a985d2da, rsp = 0x1104e18eaa88, rbp = 0x1104e18eaaa0 ---

Tracing command zfskern pid 21987 tid 100353 td 0xfffffe00c6587740
sched_switch() at sched_switch+0x559/frame 0xfffffe0084f7fdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe0084f7fde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084f7fe20
_sleep() at _sleep+0x2e7/frame 0xfffffe0084f7fec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0084f7fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084f7ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084f7ff30
--- trap 0xc, rip = 0x1104a985d2da, rsp = 0x1104ec568a88, rbp = 0x1104ec568aa0 ---

Tracing command zfskern pid 21987 tid 100354 td 0xfffffe00d8355560
sched_switch() at sched_switch+0x559/frame 0xfffffe0084fb6dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe0084fb6de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084fb6e20
_sleep() at _sleep+0x2e7/frame 0xfffffe0084fb6ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0084fb6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084fb6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084fb6f30
--- trap 0xc, rip = 0x1104a985d2da, rsp = 0x1104ee728a88, rbp = 0x1104ee728aa0 ---

Tracing command zfskern pid 21987 tid 100355 td 0xfffffe00d8365720
sched_switch() at sched_switch+0x559/frame 0xfffffe00971f1dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00971f1de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00971f1e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00971f1ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00971f1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00971f1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00971f1f30
--- trap 0x16, rip = 0x1104a985d2da, rsp = 0x1104b7904a88, rbp = 0x1104b7904aa0 ---

Tracing command zfskern pid 21987 tid 100356 td 0xfffffe00d83563a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00971ecdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00971ecde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00971ece20
_sleep() at _sleep+0x2e7/frame 0xfffffe00971ecec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00971ecef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00971ecf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00971ecf30
--- trap 0xc, rip = 0x1104a985d2da, rsp = 0x1104d2ff2a88, rbp = 0x1104d2ff2aa0 ---

Tracing command zfskern pid 21987 tid 100357 td 0xfffffe00d836fc80
sched_switch() at sched_switch+0x559/frame 0xfffffe00971c9c20
mi_switch() at mi_switch+0x157/frame 0xfffffe00971c9c40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00971c9c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00971c9cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe00971c9d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00971c9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00971c9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00971c9f30
--- trap 0xc, rip = 0x1104a985d2da, rsp = 0x1104bd68ca88, rbp = 0x1104bd68caa0 ---

Tracing command zfskern pid 21987 tid 100358 td 0xfffffe00d826c740
sched_switch() at sched_switch+0x559/frame 0xfffffe00d75b2dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00d75b2de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d75b2e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00d75b2ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00d75b2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d75b2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d75b2f30
--- trap 0x16, rip = 0x1104a985d2da, rsp = 0x1104da971a88, rbp = 0x1104da971aa0 ---

Tracing command zfskern pid 21987 tid 100359 td 0xfffffe00d8356ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00971d8cc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00971d8ce0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00971d8d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00971d8d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00971d8d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00971d8de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00971d8e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00971d8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00971d8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00971d8f30
--- trap 0xc, rip = 0x1104a985d2da, rsp = 0x1104e62a3a88, rbp = 0x1104e62a3aa0 ---

Tracing command aiod1 pid 21994 tid 100114 td 0xfffffe00c6586900
sched_switch() at sched_switch+0x559/frame 0xfffffe00cf3aad60
mi_switch() at mi_switch+0x157/frame 0xfffffe00cf3aad80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cf3aadc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00cf3aae00
_sleep() at _sleep+0x2ce/frame 0xfffffe00cf3aaea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00cf3aaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cf3aaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cf3aaf30
--- trap 0xc, rip = 0x2f18b872f2da, rsp = 0x2f18b6c933e8, rbp = 0x2f18b6c93400 ---

Tracing command aiod3 pid 21996 tid 100112 td 0xfffffe00c6461900
sched_switch() at sched_switch+0x559/frame 0xfffffe00cf3b4d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00cf3b4d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cf3b4dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00cf3b4e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00cf3b4ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00cf3b4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cf3b4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cf3b4f30
--- trap 0xc, rip = 0x7cf6ab5e31a, rsp = 0x7cf67264978, rbp = 0x7cf672649a0 ---

Tracing command ng_queue pid 36596 tid 100101 td 0xfffffe00c6583000
sched_switch() at sched_switch+0x559/frame 0xfffffe00d75a3d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00d75a3da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d75a3de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00d75a3e80
ngthread() at ngthread+0x88/frame 0xfffffe00d75a3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d75a3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d75a3f30
--- trap 0xc, rip = 0xbef0727e2da, rsp = 0xbef04b92968, rbp = 0xbef04b92a60 ---

Tracing command ng_queue pid 36596 tid 102283 td 0xfffffe00d9905900
sched_switch() at sched_switch+0x559/frame 0xfffffe00d9a9dd80
mi_switch() at mi_switch+0x157/frame 0xfffffe00d9a9dda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d9a9dde0
_sleep() at _sleep+0x2e7/frame 0xfffffe00d9a9de80
ngthread() at ngthread+0x88/frame 0xfffffe00d9a9def0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d9a9df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9a9df30
--- trap 0x16, rip = 0x3bcc2dc0c2da, rsp = 0x3bcc37b5ca88, rbp = 0x3bcc37b5caa0 ---

Tracing command aiod21 pid 85937 tid 100933 td 0xfffffe00d98c9000
sched_switch() at sched_switch+0x559/frame 0xfffffe00d8431d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00d8431d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d8431dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00d8431e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00d8431ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00d8431ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d8431f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d8431f30
--- trap 0xc, rip = 0xadedbd282da, rsp = 0xadedaee1c48, rbp = 0xadedaee1c60 ---

Tracing command aiod22 pid 85938 tid 100813 td 0xfffffe00d9936720
sched_switch() at sched_switch+0x559/frame 0xfffffe00d9b8fd60
mi_switch() at mi_switch+0x157/frame 0xfffffe00d9b8fd80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d9b8fdc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00d9b8fe00
_sleep() at _sleep+0x2ce/frame 0xfffffe00d9b8fea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00d9b8fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d9b8ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9b8ff30
--- trap 0xc, rip = 0x3953be3552da, rsp = 0x3953ba5a2b78, rbp = 0x3953ba5a2b90 ---

Tracing command soaiod1 pid 85955 tid 100831 td 0xfffffe00d989c560
sched_switch() at sched_switch+0x559/frame 0xfffffe0097192d70
mi_switch() at mi_switch+0x157/frame 0xfffffe0097192d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0097192dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0097192e10
_sleep() at _sleep+0x2ce/frame 0xfffffe0097192eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe0097192ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0097192f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0097192f30
--- trap 0xc, rip = 0x11339f35b2da, rsp = 0x11339da041b8, rbp = 0x11339da041d0 ---

Tracing command soaiod2 pid 85956 tid 100116 td 0xfffffe00c6585ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00cf3a0d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00cf3a0d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cf3a0dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00cf3a0e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00cf3a0eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00cf3a0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cf3a0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cf3a0f30
--- trap 0xc, rip = 0x123d89582da, rsp = 0x123d52076f8, rbp = 0x123d5207710 ---

Tracing command soaiod3 pid 85957 tid 100815 td 0xfffffe00d8371900
sched_switch() at sched_switch+0x559/frame 0xfffffe0084fedd70
mi_switch() at mi_switch+0x157/frame 0xfffffe0084fedd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084feddd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0084fede10
_sleep() at _sleep+0x2ce/frame 0xfffffe0084fedeb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe0084fedef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084fedf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084fedf30
--- trap 0xc, rip = 0x2b458ec572da, rsp = 0x2b458c199f18, rbp = 0x2b458c199f30 ---

Tracing command soaiod4 pid 85958 tid 100853 td 0xfffffe00d9894740
sched_switch() at sched_switch+0x559/frame 0xfffffe00d9b44d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00d9b44d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d9b44dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00d9b44e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00d9b44eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00d9b44ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d9b44f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9b44f30
--- trap 0xc, rip = 0x12ead8aba2da, rsp = 0x12ead4c90038, rbp = 0x12ead4c90050 ---

Tracing command g_mp_kt pid 28668 tid 100879 td 0xfffffe00d834e900
sched_switch() at sched_switch+0x559/frame 0xfffffe00d8454da0
mi_switch() at mi_switch+0x157/frame 0xfffffe00d8454dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00d8454e00
_sleep() at _sleep+0x2e7/frame 0xfffffe00d8454ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe00d8454ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00d8454f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d8454f30
--- trap 0xc, rip = 0x31ed4075e2da, rsp = 0x31ed3edde978, rbp = 0x31ed3eddea70 ---
db:0:kdb.enter.panic> dump

Dump failed. Partition too small (about 1471MB were needed this time).
Cannot dump: unknown error (error=7).
db:0:kdb.enter.panic> reset
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-20420' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-zfs-tests.sh
x ./run-kyua.sh
x ./disable-notyet-tests.sh
x ./auto-shutdown
x ./run.sh
x ./disable-dtrace-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins7627110488780075545.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '89d57b94d776877f77cc04752e449dac57a14618', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../89d57b94d776877f77cc04752e449dac57a14618/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address