FreeBSD-main-amd64-test - Build #19881 - Still Failing

FreeBSD-main-amd64-test - Build #19881 (afc5ab870d8661a7dacdbd65554c2f3533aecb8d) - Still Failing

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19881/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19881/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19881/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

9109192197166e72ce0c04e5f04fc82f35a82f40 by imp:
kqueue: Define older kqueue event types better

afc5ab870d8661a7dacdbd65554c2f3533aecb8d by imp:
Basic signal tests: Can we deliver a signal?



The end of the build log:

[...truncated 4.25 MB...]
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f3df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f3df30
--- trap 0xc, rip = 0x25cabf5e4d9a, rsp = 0x13a9ef48, rbp = 0x13a9ef60 ---

Tracing command zfskern pid 6722 tid 100603 td 0xfffffe00a6aa9740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f38dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f38de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f38e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6f38ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00a6f38ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f38f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f38f30
--- trap 0xc, rip = 0x25cabf5e4d9a, rsp = 0x13c9ff48, rbp = 0x13c9ff60 ---

Tracing command zfskern pid 6722 tid 100604 td 0xfffffe009dbc2020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6f33c20
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6f33c40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6f33c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a6f33cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe00a6f33d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00a6f33ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6f33f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6f33f30
--- trap 0xc, rip = 0x25cabf5e4d9a, rsp = 0x13ea0f48, rbp = 0x13ea0f60 ---

Tracing command zfskern pid 6722 tid 100605 td 0xfffffe00a6ab7560
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9d3dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9d3de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9d3e20
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9d3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe009d9d3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9d3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9d3f30
--- trap 0xc, rip = 0x3acc7e25ad9a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command zfskern pid 6722 tid 100606 td 0xfffffe00a6aa5e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c1dcc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c1dce0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c1dd20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00a6c1dd70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00a6c1dd80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00a6c1dde0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00a6c1de00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00a6c1def0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c1df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c1df30
--- trap 0xc, rip = 0x1bfb53e58d9a, rsp = 0x202f48, rbp = 0x202f60 ---

Tracing command aiod1 pid 6729 tid 100110 td 0xfffffe009db77ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00773e3d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00773e3d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773e3dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00773e3e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00773e3ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00773e3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773e3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773e3f30
--- trap 0xc, rip = 0x1e467898ad9a, rsp = 0x7fffff6f5ab8, rbp = 0x7fffff6f5ad0 ---

Tracing command aiod2 pid 6730 tid 100120 td 0xfffffe00775b43a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5db1d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5db1d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5db1dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5db1e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5db1ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5db1ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5db1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5db1f30
--- trap 0xc, rip = 0x83c84da4d9a, rsp = 0x7fffffbc1bc8, rbp = 0x7fffffbc1be0 ---

Tracing command auditd pid 673 tid 100091 td 0xfffffe00775a8560
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b4db50
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b4db70
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b4dbb0
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b4dc00
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b4dc10
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b4dcb0
audit_read() at audit_read+0x9c/frame 0xfffffe0063b4dce0
devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0063b4dd40
dofileread() at dofileread+0x81/frame 0xfffffe0063b4dd90
sys_read() at sys_read+0xc0/frame 0xfffffe0063b4de00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b4df30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b4df30
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x2ce3da07215a, rsp = 0x7fffffb75448, rbp = 0x7fffffb75490 ---

Tracing command sh pid 725 tid 100090 td 0xfffffe00775a8c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b52a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b52a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b52a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b52ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b52ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b52b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b52c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b52e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b52f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b52f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x324b1d9fd33a, rsp = 0x7fffff92df88, rbp = 0x7fffff92e040 ---

Tracing command sh pid 738 tid 100089 td 0xfffffe00775a93a0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b57a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b57a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b57a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b57ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b57ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b57b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b57c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b57e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b57f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b57f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x22063888133a, rsp = 0x7ffffff50378, rbp = 0x7ffffff50430 ---

Tracing command sh pid 745 tid 100107 td 0xfffffe009db79020
sched_switch() at sched_switch+0x559/frame 0xfffffe00773f2a10
mi_switch() at mi_switch+0x157/frame 0xfffffe00773f2a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773f2a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00773f2ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00773f2ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00773f2b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00773f2c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00773f2e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00773f2f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00773f2f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x34d61f2933a, rsp = 0x7fffff203be8, rbp = 0x7fffff203ca0 ---

Tracing command kyua pid 746 tid 100096 td 0xfffffe00775b3560
sched_switch() at sched_switch+0x559/frame 0xfffffe0063ba2a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063ba2a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063ba2a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063ba2ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063ba2ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063ba2b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063ba2c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063ba2e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063ba2f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063ba2f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0xe51e718f33a, rsp = 0x7fffffb82ce8, rbp = 0x7fffffb82d20 ---

Tracing command KTLS pid 31760 tid 101251 td 0xfffffe00a6acc3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe009d9d8d00
mi_switch() at mi_switch+0x157/frame 0xfffffe009d9d8d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe009d9d8d60
_sleep() at _sleep+0x2e7/frame 0xfffffe009d9d8e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe009d9d8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe009d9d8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d9d8f30
--- trap 0xc, rip = 0x3a1f82f34d9a, rsp = 0x7fffffd084c8, rbp = 0x7fffffd085c0 ---

Tracing command KTLS pid 31760 tid 102179 td 0xfffffe00a8e64ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6d03d00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6d03d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6d03d60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6d03e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a6d03ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6d03f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6d03f30
--- trap 0x16, rip = 0x33b8da08ad9a, rsp = 0x805a88, rbp = 0x805aa0 ---

Tracing command KTLS pid 31760 tid 102180 td 0xfffffe00a8e651e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6cead40
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6cead60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6ceada0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6ceae40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe00a6ceaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6ceaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6ceaf30
--- trap 0x16, rip = 0x33b8da08ad9a, rsp = 0xa06a88, rbp = 0xa06aa0 ---

Tracing command kyua pid 85259 tid 102296 td 0xfffffe00a9cc5000 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00097cbe00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00097cbe10
trap() at trap+0x56/frame 0xfffffe00097cbf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00097cbf20
--- trap 0x13, rip = 0xffffffff80c8a00d, rsp = 0xfffffe00a6e89550, rbp = 0xfffffe00a6e89700 ---
witness_checkorder() at witness_checkorder+0x29d/frame 0xfffffe00a6e89700
_rw_wlock_cookie() at _rw_wlock_cookie+0x67/frame 0xfffffe00a6e89740
pmap_remove_pages() at pmap_remove_pages+0x620/frame 0xfffffe00a6e89890
exec_new_vmspace() at exec_new_vmspace+0x209/frame 0xfffffe00a6e89900
exec_elf64_imgact() at exec_elf64_imgact+0x877/frame 0xfffffe00a6e899f0
kern_execve() at kern_execve+0x70d/frame 0xfffffe00a6e89d80
sys_execve() at sys_execve+0x5a/frame 0xfffffe00a6e89e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00a6e89f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00a6e89f30
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0xe51e718c41a, rsp = 0x7fffffb82858, rbp = 0x7fffffb82860 ---

Tracing command ng_queue pid 36690 tid 101284 td 0xfffffe00a6af0560
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5de8d80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5de8da0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5de8de0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a5de8e80
ngthread() at ngthread+0x88/frame 0xfffffe00a5de8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5de8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5de8f30
--- trap 0xc, rip = 0x2b5e4a804d9a, rsp = 0x7fffff9c43a8, rbp = 0x7fffff9c44a0 ---

Tracing command ng_queue pid 36690 tid 102211 td 0xfffffe00a6b401e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6fbfd80
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6fbfda0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6fbfde0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6fbfe80
ngthread() at ngthread+0x88/frame 0xfffffe00a6fbfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6fbff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6fbff30
--- trap 0x16, rip = 0x29116e266d9a, rsp = 0x7fffdf54aa88, rbp = 0x7fffdf54aaa0 ---
db:0:kdb.enter.panic> dump
Dumping 479 out of 4058 MB:..4%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19881' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-zfs-tests.sh
x ./run.sh
x ./auto-shutdown
x ./disable-dtrace-tests.sh
x ./run-kyua.sh
x ./disable-notyet-tests.sh
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins8751512234260072855.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'afc5ab870d8661a7dacdbd65554c2f3533aecb8d', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../afc5ab870d8661a7dacdbd65554c2f3533aecb8d/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address