FreeBSD-main-amd64-test - Build #19778 - Still Failing

FreeBSD-main-amd64-test - Build #19778 (d69538633815127b84fe7472cdc44f4b9e6e7780) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19778/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19778/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19778/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

26cbd0028c508787e1d88361f345ac707acabce5 by rrs:
tcp: Rack may still calculate long RTT on persists probes.

d69538633815127b84fe7472cdc44f4b9e6e7780 by rrs:
Add in the commit revision to the UPDATING file for the CC changes



The end of the build log:

[...truncated 1.01 MB...]
Tracing command zfskern pid 63356 tid 100786 td 0xfffffe00b2fc5c80
sched_switch() at sched_switch+0x578/frame 0xfffffe00b36dbdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b36dbde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b36dbe20
_sleep() at _sleep+0x29e/frame 0xfffffe00b36dbec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b36dbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b36dbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b36dbf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdc1dea88, rbp = 0x7fffdc1deaa0 ---

Tracing command zfskern pid 63356 tid 100787 td 0xfffffe00b2fcf3a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b31dadc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b31dade0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b31dae20
_sleep() at _sleep+0x29e/frame 0xfffffe00b31daec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b31daef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b31daf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b31daf30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffde1eea88, rbp = 0x7fffde1eeaa0 ---

Tracing command zfskern pid 63356 tid 100788 td 0xfffffe00b2fa2900
sched_switch() at sched_switch+0x578/frame 0xfffffe00b373fdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b373fde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b373fe20
_sleep() at _sleep+0x29e/frame 0xfffffe00b373fec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b373fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b373ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b373ff30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdb1d6a88, rbp = 0x7fffdb1d6aa0 ---

Tracing command zfskern pid 63356 tid 100789 td 0xfffffe00b2fabc80
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3663dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3663de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3663e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3663ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3663ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3663f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3663f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdcbe3a88, rbp = 0x7fffdcbe3aa0 ---

Tracing command zfskern pid 63356 tid 100790 td 0xfffffe00b2f5c3a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3730dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3730de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3730e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3730ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3730ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3730f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3730f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdabd3a88, rbp = 0x7fffdabd3aa0 ---

Tracing command zfskern pid 63356 tid 100791 td 0xfffffe00b2f5e740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3627dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3627de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3627e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3627ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3627ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3627f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3627f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdcde4a88, rbp = 0x7fffdcde4aa0 ---

Tracing command zfskern pid 63356 tid 100792 td 0xfffffe00b2f5a000
sched_switch() at sched_switch+0x578/frame 0xfffffe00b36e5dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b36e5de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b36e5e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b36e5ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b36e5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b36e5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b36e5f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdc5e0a88, rbp = 0x7fffdc5e0aa0 ---

Tracing command zfskern pid 63356 tid 100793 td 0xfffffe00b2fd01e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3194dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3194de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3194e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3194ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3194ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3194f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3194f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdcfe5a88, rbp = 0x7fffdcfe5aa0 ---

Tracing command zfskern pid 63356 tid 100794 td 0xfffffe00b2f2c740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3672dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3672de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3672e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3672ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3672ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3672f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3672f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdd9eaa88, rbp = 0x7fffdd9eaaa0 ---

Tracing command zfskern pid 63356 tid 100795 td 0xfffffe00b2fae020
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3726dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3726de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3726e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3726ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3726ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3726f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3726f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffda7d1a88, rbp = 0x7fffda7d1aa0 ---

Tracing command zfskern pid 63356 tid 100796 td 0xfffffe00b2facac0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3712dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3712de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3712e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b3712ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b3712ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3712f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3712f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffd9fcda88, rbp = 0x7fffd9fcdaa0 ---

Tracing command zfskern pid 63356 tid 100797 td 0xfffffe00b2fb6900
sched_switch() at sched_switch+0x578/frame 0xfffffe00b31b2dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b31b2de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b31b2e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b31b2ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b31b2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b31b2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b31b2f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffde5f0a88, rbp = 0x7fffde5f0aa0 ---

Tracing command zfskern pid 63356 tid 100798 td 0xfffffe00b2f2c020
sched_switch() at sched_switch+0x578/frame 0xfffffe00b31f8dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b31f8de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b31f8e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b31f8ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b31f8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b31f8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b31f8f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdd1e6a88, rbp = 0x7fffdd1e6aa0 ---

Tracing command zfskern pid 63356 tid 100799 td 0xfffffe00b2fcd000
sched_switch() at sched_switch+0x578/frame 0xfffffe00b36a4dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b36a4de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b36a4e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b36a4ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b36a4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b36a4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b36a4f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdbfdda88, rbp = 0x7fffdbfddaa0 ---

Tracing command zfskern pid 63356 tid 100800 td 0xfffffe00b2fb7740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3158c20
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3158c40
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3158c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3158cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe00b3158d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00b3158ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3158f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3158f30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffddbeba88, rbp = 0x7fffddbebaa0 ---

Tracing command zfskern pid 63356 tid 100801 td 0xfffffe00b2fcfac0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b322fdc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b322fde0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b322fe20
_sleep() at _sleep+0x29e/frame 0xfffffe00b322fec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b322fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b322ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b322ff30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdf3f7a88, rbp = 0x7fffdf3f7aa0 ---

Tracing command zfskern pid 63356 tid 100802 td 0xfffffe00b2fcec80
sched_switch() at sched_switch+0x578/frame 0xfffffe00b36efcc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b36efce0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b36efd20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00b36efd70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00b36efd80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00b36efde0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00b36efe00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00b36efef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b36eff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b36eff30
--- trap 0xc, rip = 0x801396bba, rsp = 0x7fffdedf4a88, rbp = 0x7fffdedf4aa0 ---

Tracing command KTLS pid 94152 tid 100132 td 0xfffffe00b2e65000
sched_switch() at sched_switch+0x578/frame 0xfffffe0063bbcd00
mi_switch() at mi_switch+0x155/frame 0xfffffe0063bbcd20
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063bbcd60
_sleep() at _sleep+0x29e/frame 0xfffffe0063bbce00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe0063bbcef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bbcf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bbcf30
--- trap 0xc, rip = 0x8011a6bba, rsp = 0x7fffffffea08, rbp = 0x7fffffffea20 ---

Tracing command KTLS pid 94152 tid 101772 td 0xfffffe00b2fd1020
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3207d00
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3207d20
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3207d60
_sleep() at _sleep+0x29e/frame 0xfffffe00b3207e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00b3207ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3207f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3207f30
--- trap 0xc, rip = 0x8011ecbba, rsp = 0x7fffda9d2f48, rbp = 0x7fffda9d2f60 ---

Tracing command KTLS pid 94152 tid 101773 td 0xfffffe00b2ffb1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b31e4d40
mi_switch() at mi_switch+0x155/frame 0xfffffe00b31e4d60
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b31e4da0
_sleep() at _sleep+0x29e/frame 0xfffffe00b31e4e40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe00b31e4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b31e4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b31e4f30
--- trap 0xc, rip = 0x8011ecbba, rsp = 0x7fffdc3dff48, rbp = 0x7fffdc3dff60 ---
db:0:kdb.enter.panic> dump
Dumping 433 out of 4058 MB:..4%..12%..23%..34%..41%..52%..63%..71%..82%..93%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19778' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-notyet-tests.sh
x ./run-kyua.sh
x ./auto-shutdown
x ./disable-zfs-tests.sh
x ./disable-dtrace-tests.sh
x ./run.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins7938882464888716941.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'd69538633815127b84fe7472cdc44f4b9e6e7780', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../d69538633815127b84fe7472cdc44f4b9e6e7780/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address