FreeBSD-main-amd64-test - Build #19781 - Still Failing

FreeBSD-main-amd64-test - Build #19781 (e38717c1282f8bc8b16389839bea015359413df8) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19781/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19781/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19781/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

e38717c1282f8bc8b16389839bea015359413df8 by mckusick:
Fix regression to verbose behavior introduced in 68bff4a07e.



The end of the build log:

[...truncated 4.16 MB...]
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6e57dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6e57de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6e57e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a6e57ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6e57ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6e57f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6e57f30
--- trap 0xc, rip = 0x801553bba, rsp = 0x7fffdf9faa88, rbp = 0x7fffdf9faaa0 ---

Tracing command zfskern pid 59347 tid 100846 td 0xfffffe00a6a86000
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6e89dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6e89de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6e89e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a6e89ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6e89ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6e89f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6e89f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffde9f2a88, rbp = 0x7fffde9f2aa0 ---

Tracing command zfskern pid 59347 tid 100847 td 0xfffffe00a6bfd900
sched_switch() at sched_switch+0x578/frame 0xfffffe00a72a9dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a72a9de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a72a9e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a72a9ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a72a9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a72a9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a72a9f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd7fbda88, rbp = 0x7fffd7fbdaa0 ---

Tracing command zfskern pid 59347 tid 100848 td 0xfffffe00a7019740
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7376dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7376de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7376e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a7376ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a7376ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7376f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7376f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffdf7f9a88, rbp = 0x7fffdf7f9aa0 ---

Tracing command zfskern pid 59347 tid 100849 td 0xfffffe00a7020000
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6cccdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6cccde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6ccce20
_sleep() at _sleep+0x29e/frame 0xfffffe00a6cccec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6cccef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6cccf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6cccf30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd87c1a88, rbp = 0x7fffd87c1aa0 ---

Tracing command zfskern pid 59347 tid 100850 td 0xfffffe00a7015000
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6d44dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6d44de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6d44e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a6d44ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6d44ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6d44f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6d44f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd81bea88, rbp = 0x7fffd81beaa0 ---

Tracing command zfskern pid 59347 tid 100851 td 0xfffffe00a6a86720
sched_switch() at sched_switch+0x578/frame 0xfffffe00a736cdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a736cde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a736ce20
_sleep() at _sleep+0x29e/frame 0xfffffe00a736cec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a736cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a736cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a736cf30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffdb3d7a88, rbp = 0x7fffdb3d7aa0 ---

Tracing command zfskern pid 59347 tid 100852 td 0xfffffe00a6bfd1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6e20dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6e20de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6e20e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a6e20ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6e20ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6e20f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6e20f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd77b9a88, rbp = 0x7fffd77b9aa0 ---

Tracing command zfskern pid 59347 tid 100853 td 0xfffffe00a6a88ac0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7371dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7371de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7371e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a7371ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a7371ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7371f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7371f30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffda7d1a88, rbp = 0x7fffda7d1aa0 ---

Tracing command zfskern pid 59347 tid 100854 td 0xfffffe00a6bfa720
sched_switch() at sched_switch+0x578/frame 0xfffffe00a733fdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a733fde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a733fe20
_sleep() at _sleep+0x29e/frame 0xfffffe00a733fec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a733fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a733ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a733ff30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd7bbba88, rbp = 0x7fffd7bbbaa0 ---

Tracing command zfskern pid 59347 tid 100855 td 0xfffffe00a700ce40
sched_switch() at sched_switch+0x578/frame 0xfffffe00a72b3dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a72b3de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a72b3e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a72b3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a72b3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a72b3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a72b3f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd91c6a88, rbp = 0x7fffd91c6aa0 ---

Tracing command zfskern pid 59347 tid 100856 td 0xfffffe00a6a8a020
sched_switch() at sched_switch+0x578/frame 0xfffffe00a72f4dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a72f4de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a72f4e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a72f4ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a72f4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a72f4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a72f4f30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffdc9e2a88, rbp = 0x7fffdc9e2aa0 ---

Tracing command zfskern pid 59347 tid 100857 td 0xfffffe00a6b19ac0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a732bdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a732bde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a732be20
_sleep() at _sleep+0x29e/frame 0xfffffe00a732bec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a732bef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a732bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a732bf30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffdbfdda88, rbp = 0x7fffdbfddaa0 ---

Tracing command zfskern pid 59347 tid 100858 td 0xfffffe00a6bfe020
sched_switch() at sched_switch+0x578/frame 0xfffffe00a733adc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a733ade0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a733ae20
_sleep() at _sleep+0x29e/frame 0xfffffe00a733aec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a733aef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a733af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a733af30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffd93c7a88, rbp = 0x7fffd93c7aa0 ---

Tracing command zfskern pid 59347 tid 100859 td 0xfffffe00a7010020
sched_switch() at sched_switch+0x578/frame 0xfffffe00a734edc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a734ede0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a734ee20
_sleep() at _sleep+0x29e/frame 0xfffffe00a734eec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a734eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a734ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a734ef30
--- trap 0x16, rip = 0x8013d4bba, rsp = 0x7fffd9dcca88, rbp = 0x7fffd9dccaa0 ---

Tracing command zfskern pid 59347 tid 100860 td 0xfffffe00a6bf33a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a72eadc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a72eade0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a72eae20
_sleep() at _sleep+0x29e/frame 0xfffffe00a72eaec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a72eaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a72eaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a72eaf30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffdd3e7a88, rbp = 0x7fffdd3e7aa0 ---

Tracing command zfskern pid 59347 tid 100861 td 0xfffffe00a6bfb560
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6e2fc20
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6e2fc40
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6e2fc80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6e2fcc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe00a6e2fd40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00a6e2fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6e2ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6e2ff30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd8bc3a88, rbp = 0x7fffd8bc3aa0 ---

Tracing command zfskern pid 59347 tid 100862 td 0xfffffe00a700f1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a7308dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a7308de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a7308e20
_sleep() at _sleep+0x29e/frame 0xfffffe00a7308ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a7308ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a7308f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a7308f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd85c0a88, rbp = 0x7fffd85c0aa0 ---

Tracing command zfskern pid 59347 tid 100863 td 0xfffffe00a6b193a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00a6dc6cc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00a6dc6ce0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00a6dc6d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a6dc6d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a6dc6d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00a6dc6de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00a6dc6e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00a6dc6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6dc6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6dc6f30
--- trap 0xc, rip = 0x8013d4bba, rsp = 0x7fffd79baa88, rbp = 0x7fffd79baaa0 ---

Tracing command aiod3 pid 59356 tid 100144 td 0xfffffe00a69bd3a0
sched_switch() at sched_switch+0x578/frame 0xfffffe009d94ed60
mi_switch() at mi_switch+0x155/frame 0xfffffe009d94ed80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe009d94edc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe009d94ee00
_sleep() at _sleep+0x282/frame 0xfffffe009d94eea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe009d94eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d94ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d94ef30
--- trap 0xc, rip = 0x8011fabba, rsp = 0x7fffffffea48, rbp = 0x7fffffffea60 ---
db:0:kdb.enter.panic> dump
Dumping 437 out of 4058 MB:..4%..11%..22%..33%..41%..52%..63%..74%..81%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19781' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-dtrace-tests.sh
x ./disable-zfs-tests.sh
x ./run.sh
x ./run-kyua.sh
x ./disable-notyet-tests.sh
x ./auto-shutdown
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins4190796161059081666.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'e38717c1282f8bc8b16389839bea015359413df8', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../e38717c1282f8bc8b16389839bea015359413df8/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address