FreeBSD-main-amd64-test - Build #19862 - Still Failing

From: <>
Date: Sat, 20 Nov 2021 10:44:46 UTC
FreeBSD-main-amd64-test - Build #19862 (0c54fe172ad365e7e60d6249484a7579c18b7d2d) - Still Failing

Build information:
Full change log:
Full build log:

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

0c54fe172ad365e7e60d6249484a7579c18b7d2d by eugen:
rc.d/rctl: unbreak for distinct /usr filesystem

The end of the build log:

[...truncated 3.94 MB...]
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command auditd pid 673 tid 100087 td 0xfffffe00775aa1e0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b61b50
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b61b70
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b61bb0
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b61c00
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b61c10
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b61cb0
audit_read() at audit_read+0x9c/frame 0xfffffe0063b61ce0
devfs_read_f() at devfs_read_f+0xe5/frame 0xfffffe0063b61d40
dofileread() at dofileread+0x81/frame 0xfffffe0063b61d90
sys_read() at sys_read+0xc0/frame 0xfffffe0063b61e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b61f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b61f30
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x3da156e8f15a, rsp = 0x7ffffffdb2a8, rbp = 0x7ffffffdb2f0 ---

Tracing command sh pid 725 tid 100090 td 0xfffffe00775a8c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b52a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b52a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b52a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b52ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b52ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b52b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b52c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b52e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b52f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b52f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1796157ce33a, rsp = 0x7fffff5a9328, rbp = 0x7fffff5a93e0 ---

Tracing command sh pid 738 tid 100093 td 0xfffffe00775a7720
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b43a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b43a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b43a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b43ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b43ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b43b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b43c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b43e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b43f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b43f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x1b947bf5133a, rsp = 0x7fffff4a9688, rbp = 0x7fffff4a9740 ---

Tracing command sh pid 745 tid 100098 td 0xfffffe00775b2720
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b98a10
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b98a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b98a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063b98ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063b98ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe0063b98b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063b98c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063b98e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe0063b98f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063b98f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x15437092433a, rsp = 0x7fffffeb1778, rbp = 0x7fffffeb1830 ---

Tracing command kyua pid 746 tid 100107 td 0xfffffe009db79020
sched_switch() at sched_switch+0x559/frame 0xfffffe00773f2a10
mi_switch() at mi_switch+0x157/frame 0xfffffe00773f2a30
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773f2a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00773f2ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00773f2ad0
_sleep() at _sleep+0x2dd/frame 0xfffffe00773f2b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00773f2c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00773f2e00
amd64_syscall() at amd64_syscall+0x6b4/frame 0xfffffe00773f2f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00773f2f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x29dadc3fc33a, rsp = 0x7fffff8d3808, rbp = 0x7fffff8d3840 ---

Tracing command g_mp_kt pid 60354 tid 100129 td 0xfffffe009dbb8000
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bb7da0
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bb7dc0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bb7e00
_sleep() at _sleep+0x2e7/frame 0xfffffe0063bb7ea0
g_multipath_kt() at g_multipath_kt+0x275/frame 0xfffffe0063bb7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bb7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bb7f30
--- trap 0xc, rip = 0xd78affd6d9a, rsp = 0x7fffffc97258, rbp = 0x7fffffc97350 ---

Tracing command aiod6 pid 17639 tid 100120 td 0xfffffe009dbba3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bd0d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bd0d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bd0dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063bd0e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063bd0ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063bd0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bd0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bd0f30
--- trap 0xc, rip = 0x2e828cc4bd9a, rsp = 0x7fffff991658, rbp = 0x7fffff991670 ---

Tracing command aiod16 pid 17649 tid 100092 td 0xfffffe00775a7e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b48d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b48d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b48dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b48e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b48ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b48ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b48f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b48f30
--- trap 0xc, rip = 0x2d2d5b4c5d9a, rsp = 0x7fffff66fd48, rbp = 0x7fffff66fe40 ---

Tracing command aiod18 pid 17651 tid 100097 td 0xfffffe00775b2e40
sched_switch() at sched_switch+0x559/frame 0xfffffe0063b9dd60
mi_switch() at mi_switch+0x157/frame 0xfffffe0063b9dd80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063b9ddc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063b9de00
_sleep() at _sleep+0x2ce/frame 0xfffffe0063b9dea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063b9def0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063b9df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063b9df30
--- trap 0xc, rip = 0x10751ef08d9a, rsp = 0x7fffff75d698, rbp = 0x7fffff75d790 ---

Tracing command aiod28 pid 17661 tid 100126 td 0xfffffe00a6a14e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5da7d60
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5da7d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5da7dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5da7e00
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5da7ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00a5da7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5da7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5da7f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command soaiod1 pid 17672 tid 100133 td 0xfffffe00a6a18020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5df2d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5df2d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5df2dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5df2e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5df2eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5df2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5df2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5df2f30
--- trap 0xc, rip = 0x2536c567cd9a, rsp = 0x7fffffecd7d8, rbp = 0x7fffffecd7f0 ---

Tracing command soaiod2 pid 17673 tid 100132 td 0xfffffe00a6a18740
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5df7d70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5df7d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5df7dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5df7e10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5df7eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5df7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5df7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5df7f30
--- trap 0xc, rip = 0xf577cbf3d9a, rsp = 0x7fffff51f558, rbp = 0x7fffff51f570 ---

Tracing command soaiod3 pid 17674 tid 100131 td 0xfffffe00a6a1d000
sched_switch() at sched_switch+0x559/frame 0xfffffe00a5dfcd70
mi_switch() at mi_switch+0x157/frame 0xfffffe00a5dfcd90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a5dfcdd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00a5dfce10
_sleep() at _sleep+0x2ce/frame 0xfffffe00a5dfceb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe00a5dfcef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a5dfcf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a5dfcf30
--- trap 0xc, rip = 0x1f46796d5d9a, rsp = 0x7fffff79fe88, rbp = 0x7fffff79fea0 ---

Tracing command soaiod4 pid 17675 tid 100130 td 0xfffffe009db79740
sched_switch() at sched_switch+0x559/frame 0xfffffe0063bb2d70
mi_switch() at mi_switch+0x157/frame 0xfffffe0063bb2d90
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0063bb2dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063bb2e10
_sleep() at _sleep+0x2ce/frame 0xfffffe0063bb2eb0
soaio_kproc_loop() at soaio_kproc_loop+0x125/frame 0xfffffe0063bb2ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063bb2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063bb2f30
--- trap 0xc, rip = 0xe0d0a3c4d9a, rsp = 0x7fffff4bbe28, rbp = 0x7fffff4bbe40 ---

Tracing command atf-sh pid 16188 tid 101209 td 0xfffffe00ab529720 (CPU 1)
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00097cbe00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe00097cbe10
trap() at trap+0x56/frame 0xfffffe00097cbf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00097cbf20
--- trap 0x13, rip = 0x261b016c2630, rsp = 0x7fffff5826d0, rbp = 0x7fffff582b50 ---

Tracing command KTLS pid 63318 tid 100110 td 0xfffffe009db77ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00773e3d00
mi_switch() at mi_switch+0x157/frame 0xfffffe00773e3d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00773e3d60
_sleep() at _sleep+0x2e7/frame 0xfffffe00773e3e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00773e3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00773e3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00773e3f30
--- trap 0xc, rip = 0x14a9c82b8d9a, rsp = 0x7ffffffbd878, rbp = 0x7ffffffbd930 ---

Tracing command KTLS pid 63318 tid 100833 td 0xfffffe00a6a3d020
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c04d00
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c04d20
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c04d60
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c04e00
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00a6c04ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c04f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c04f30
--- trap 0x16, rip = 0x1433c81b2d9a, rsp = 0x805a88, rbp = 0x805aa0 ---

Tracing command KTLS pid 63318 tid 100834 td 0xfffffe009dbbb900
sched_switch() at sched_switch+0x559/frame 0xfffffe00a6c0ed40
mi_switch() at mi_switch+0x157/frame 0xfffffe00a6c0ed60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00a6c0eda0
_sleep() at _sleep+0x2e7/frame 0xfffffe00a6c0ee40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe00a6c0eef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00a6c0ef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6c0ef30
--- trap 0x16, rip = 0x1433c81b2d9a, rsp = 0x604a88, rbp = 0x604aa0 ---
db:0:kdb.enter.panic> dump
Dumping 485 out of 4058 MB:..4%..14%..24%..33%..43%..53%..63%..73%..83%..93%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19862' --destroy
+ sh -ex freebsd-ci/scripts/test/
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./
x ./
x ./auto-shutdown
x ./
x ./
x ./
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/
+ ./freebsd-ci/artifact/
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '0c54fe172ad365e7e60d6249484a7579c18b7d2d', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../0c54fe172ad365e7e60d6249484a7579c18b7d2d/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address