FreeBSD-main-amd64-test - Build #19780 - Still Failing

FreeBSD-main-amd64-test - Build #19780 (811d05449bea1b3a6f0206afb4866623569071c7) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19780/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19780/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19780/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

e4bdb6857aea466ce4af48c0bd05a214460fe313 by markj:
vm_page: Handle VM_ALLOC_NORECLAIM in the contiguous page allocator

811d05449bea1b3a6f0206afb4866623569071c7 by markj:
vm_page_alloc.9: Document VM_ALLOC_NORECLAIM



The end of the build log:

[...truncated 1.04 MB...]
sched_switch() at sched_switch+0x578/frame 0xfffffe00b20ffdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b20ffde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b20ffe20
_sleep() at _sleep+0x29e/frame 0xfffffe00b20ffec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b20ffef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b20fff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b20fff30
--- trap 0xc, rip = 0x8013d0bba, rsp = 0x7fffdedf4a88, rbp = 0x7fffdedf4aa0 ---

Tracing command zfskern pid 59290 tid 100341 td 0xfffffe00b2ec2000
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2131dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2131de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2131e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2131ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2131ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2131f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2131f30
--- trap 0xc, rip = 0x8013d0bba, rsp = 0x7fffdf1f6a88, rbp = 0x7fffdf1f6aa0 ---

Tracing command zfskern pid 59290 tid 100342 td 0xfffffe00b2ec2e40
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2109dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2109de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2109e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2109ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2109ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2109f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2109f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdbddca88, rbp = 0x7fffdbddcaa0 ---

Tracing command zfskern pid 59290 tid 100343 td 0xfffffe00b2edae40
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bcbdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bcbde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bcbe20
_sleep() at _sleep+0x29e/frame 0xfffffe0063bcbec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0063bcbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bcbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bcbf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdc1dea88, rbp = 0x7fffdc1deaa0 ---

Tracing command zfskern pid 59290 tid 100344 td 0xfffffe00b2ede740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b20f5dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b20f5de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b20f5e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b20f5ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b20f5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b20f5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b20f5f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdc5e0a88, rbp = 0x7fffdc5e0aa0 ---

Tracing command zfskern pid 59290 tid 100345 td 0xfffffe00b2eb73a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b20c3dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b20c3de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b20c3e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b20c3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b20c3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b20c3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b20c3f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdc9e2a88, rbp = 0x7fffdc9e2aa0 ---

Tracing command zfskern pid 59290 tid 100346 td 0xfffffe00b2e22740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2177dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2177de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2177e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2177ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2177ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2177f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2177f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdadd4a88, rbp = 0x7fffdadd4aa0 ---

Tracing command zfskern pid 59290 tid 100347 td 0xfffffe00b2edcac0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b217cdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b217cde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b217ce20
_sleep() at _sleep+0x29e/frame 0xfffffe00b217cec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b217cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b217cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b217cf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdafd5a88, rbp = 0x7fffdafd5aa0 ---

Tracing command zfskern pid 59290 tid 100348 td 0xfffffe00b2ec3c80
sched_switch() at sched_switch+0x578/frame 0xfffffe00b210edc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b210ede0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b210ee20
_sleep() at _sleep+0x29e/frame 0xfffffe00b210eec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b210eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b210ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b210ef30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdcde4a88, rbp = 0x7fffdcde4aa0 ---

Tracing command zfskern pid 59290 tid 100349 td 0xfffffe00b2edd1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b20d7dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b20d7de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b20d7e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b20d7ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b20d7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b20d7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b20d7f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffddbeba88, rbp = 0x7fffddbebaa0 ---

Tracing command zfskern pid 59290 tid 100350 td 0xfffffe00b2eb81e0
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bb7dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bb7de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bb7e20
_sleep() at _sleep+0x29e/frame 0xfffffe0063bb7ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0063bb7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bb7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bb7f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdf3f7a88, rbp = 0x7fffdf3f7aa0 ---

Tracing command zfskern pid 59290 tid 100351 td 0xfffffe00b2e22020
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2113dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2113de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2113e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2113ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2113ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2113f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2113f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdd9eaa88, rbp = 0x7fffdd9eaaa0 ---

Tracing command zfskern pid 59290 tid 100352 td 0xfffffe00b2eb5000
sched_switch() at sched_switch+0x578/frame 0xfffffe00b213bdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b213bde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b213be20
_sleep() at _sleep+0x29e/frame 0xfffffe00b213bec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b213bef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b213bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b213bf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffde9f2a88, rbp = 0x7fffde9f2aa0 ---

Tracing command zfskern pid 59290 tid 100353 td 0xfffffe00b2eb9740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2195dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2195de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2195e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2195ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2195ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2195f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2195f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdf5f8a88, rbp = 0x7fffdf5f8aa0 ---

Tracing command zfskern pid 59290 tid 100354 td 0xfffffe00b2edd900
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bb2dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bb2de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bb2e20
_sleep() at _sleep+0x29e/frame 0xfffffe0063bb2ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe0063bb2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bb2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bb2f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdd3e7a88, rbp = 0x7fffdd3e7aa0 ---

Tracing command zfskern pid 59290 tid 100355 td 0xfffffe00b2e1ee40
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2145c20
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2145c40
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2145c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b2145cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe00b2145d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00b2145ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2145f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2145f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdfdfca88, rbp = 0x7fffdfdfcaa0 ---

Tracing command zfskern pid 59290 tid 100356 td 0xfffffe00b2ed0560
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2154dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2154de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2154e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2154ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2154ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2154f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2154f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffd9fcda88, rbp = 0x7fffd9fcdaa0 ---

Tracing command zfskern pid 59290 tid 100357 td 0xfffffe00a6fdc740
sched_switch() at sched_switch+0x578/frame 0xfffffe0063badcc0
mi_switch() at mi_switch+0x155/frame 0xfffffe0063badce0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063badd20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063badd70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063badd80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe0063badde0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe0063bade00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe0063badef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063badf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063badf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdd7e9a88, rbp = 0x7fffdd7e9aa0 ---

Tracing command aiod4 pid 59300 tid 100108 td 0xfffffe00a6e44020
sched_switch() at sched_switch+0x578/frame 0xfffffe00777d6d60
mi_switch() at mi_switch+0x155/frame 0xfffffe00777d6d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00777d6dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00777d6e00
_sleep() at _sleep+0x282/frame 0xfffffe00777d6ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00777d6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00777d6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00777d6f30
--- trap 0xc, rip = 0x801341bba, rsp = 0x7fffffffeaa8, rbp = 0x7fffffffeac0 ---

Tracing command g_mp_kt pid 20454 tid 100821 td 0xfffffe00b4074900
sched_switch() at sched_switch+0x578/frame 0xfffffe00b20e1da0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b20e1dc0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b20e1e00
_sleep() at _sleep+0x29e/frame 0xfffffe00b20e1ea0
g_multipath_kt() at g_multipath_kt+0x265/frame 0xfffffe00b20e1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b20e1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b20e1f30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe848, rbp = 0x7fffffffe940 ---
db:0:kdb.enter.panic> dump
Dumping 442 out of 4058 MB:..4%..11%..22%..33%..44%..51%..62%..73%..84%..91%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19780' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-dtrace-tests.sh
x ./disable-zfs-tests.sh
x ./run.sh
x ./auto-shutdown
x ./disable-notyet-tests.sh
x ./run-kyua.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins6500445871937316358.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '811d05449bea1b3a6f0206afb4866623569071c7', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../811d05449bea1b3a6f0206afb4866623569071c7/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address