FreeBSD-main-amd64-test - Build #20419 - Still Failing

From: <>
Date: Wed, 19 Jan 2022 03:17:25 UTC
FreeBSD-main-amd64-test - Build #20419 (b7ff445ffa38282daeab36ce82681ba3f54c8851) - Still Failing

Build information:
Full change log:
Full build log:

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

6596f19daa42d5b968030e08826542c704533b73 by rew:
etc/defaults/rc.conf: set default of zfskeys_enable to NO

b7ff445ffa38282daeab36ce82681ba3f54c8851 by mav:
Reduce bufdaemon/bufspacedaemon shutdown time.

The end of the build log:

[...truncated 4.25 MB...]
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00c2dcfe20
_sleep() at _sleep+0x2e7/frame 0xfffffe00c2dcfec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00c2dcfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c2dcff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2dcff30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100140 td 0xfffffe00c2ff51e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00c2dcadc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00c2dcade0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00c2dcae20
_sleep() at _sleep+0x2e7/frame 0xfffffe00c2dcaec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00c2dcaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c2dcaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2dcaf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100141 td 0xfffffe00c2ff4ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00c2dc5dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00c2dc5de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00c2dc5e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00c2dc5ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00c2dc5ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c2dc5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2dc5f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100142 td 0xfffffe00c2ff43a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00c2dc0dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00c2dc0de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00c2dc0e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00c2dc0ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00c2dc0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c2dc0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2dc0f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100143 td 0xfffffe00c2ff3c80
sched_switch() at sched_switch+0x559/frame 0xfffffe00c2dbbdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00c2dbbde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00c2dbbe20
_sleep() at _sleep+0x2e7/frame 0xfffffe00c2dbbec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00c2dbbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00c2dbbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2dbbf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100144 td 0xfffffe00cc9cb900
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1fddc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1fdde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1fde20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1fdec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1fdef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1fdf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1fdf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100145 td 0xfffffe00cc9cb1e0
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1f8dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1f8de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1f8e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1f8ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1f8ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1f8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1f8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100146 td 0xfffffe00cc9caac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1f3dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1f3de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1f3e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1f3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1f3ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1f3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1f3f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100147 td 0xfffffe00cc9ca3a0
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1eedc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1eede0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1eee20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1eeec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1eeef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1eef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1eef30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100148 td 0xfffffe00cc9c9c80
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1e9dc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1e9de0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1e9e20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1e9ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1e9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1e9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1e9f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100149 td 0xfffffe00cc9c9560
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1e4c20
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1e4c40
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1e4c80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00cb1e4cc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19a/frame 0xfffffe00cb1e4d40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00cb1e4ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1e4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1e4f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100150 td 0xfffffe00cc9c8e40
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1dfdc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1dfde0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1dfe20
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1dfec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00cb1dfef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1dff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1dff30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command zfskern pid 1319 tid 100151 td 0xfffffe00cc9c8720
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1dacc0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1dace0
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1dad20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00cb1dad70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00cb1dad80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00cb1dade0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00cb1dae00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00cb1daef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1daf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1daf30
--- trap 0, rip = 0, rsp = 0x1cd3c97daf18, rbp = 0x1cd3c95835d0 ---

Tracing command aiod1 pid 1327 tid 100095 td 0xfffffe00973e3000
sched_switch() at sched_switch+0x559/frame 0xfffffe0084f39d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0084f39d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084f39dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0084f39e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0084f39ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0084f39ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084f39f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084f39f30
--- trap 0xc, rip = 0x16e8e88f42da, rsp = 0x16e8e73eb928, rbp = 0x16e8e73eb940 ---

Tracing command aiod2 pid 1328 tid 100089 td 0xfffffe008f7b9c80
sched_switch() at sched_switch+0x559/frame 0xfffffe0084f57d60
mi_switch() at mi_switch+0x157/frame 0xfffffe0084f57d80
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084f57dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0084f57e00
_sleep() at _sleep+0x2ce/frame 0xfffffe0084f57ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0084f57ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084f57f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084f57f30
--- trap 0xc, rip = 0x61eacee92da, rsp = 0x61ea91edaf8, rbp = 0x61ea91edb10 ---

Tracing command KTLS pid 5601 tid 100118 td 0xfffffe00c2fe5ac0
sched_switch() at sched_switch+0x559/frame 0xfffffe00cb1adcf0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cb1add10
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cb1add50
_sleep() at _sleep+0x2e7/frame 0xfffffe00cb1addf0
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00cb1adef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cb1adf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cb1adf30
--- trap 0xc, rip = 0x3f72f6d372da, rsp = 0x3f72f0f6b878, rbp = 0x3f72f0f6b880 ---

Tracing command KTLS pid 5601 tid 102140 td 0xfffffe00ce1acc80
sched_switch() at sched_switch+0x559/frame 0xfffffe00cccfecf0
mi_switch() at mi_switch+0x157/frame 0xfffffe00cccfed10
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe00cccfed50
_sleep() at _sleep+0x2e7/frame 0xfffffe00cccfedf0
ktls_work_thread() at ktls_work_thread+0x19c/frame 0xfffffe00cccfeef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00cccfef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00cccfef30
--- trap 0x16, rip = 0x31ee601062da, rsp = 0x31ee6b098a88, rbp = 0x31ee6b098aa0 ---

Tracing command KTLS pid 5601 tid 102141 td 0xfffffe00cca2b560
sched_switch() at sched_switch+0x559/frame 0xfffffe0084fa7d40
mi_switch() at mi_switch+0x157/frame 0xfffffe0084fa7d60
sleepq_switch() at sleepq_switch+0x107/frame 0xfffffe0084fa7da0
_sleep() at _sleep+0x2e7/frame 0xfffffe0084fa7e40
ktls_alloc_thread() at ktls_alloc_thread+0x204/frame 0xfffffe0084fa7ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0084fa7f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084fa7f30
--- trap 0x16, rip = 0x31ee601062da, rsp = 0x31ee6b2c9a88, rbp = 0x31ee6b2c9aa0 ---

Tracing command posixshm_test pid 91823 tid 102218 td 0xfffffe00ccf41000 (CPU 1)
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0084ffb910
vpanic() at vpanic+0x1b0/frame 0xfffffe0084ffb960
panic() at panic+0x43/frame 0xfffffe0084ffb9c0
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0084ffba20
calltrap() at calltrap+0x8/frame 0xfffffe0084ffba20
--- trap 0x9, rip = 0xffffffff80f885f2, rsp = 0xfffffe0084ffbaf0, rbp = 0xfffffe0084ffbb60 ---
vm_phys_alloc_contig() at vm_phys_alloc_contig+0x232/frame 0xfffffe0084ffbb60
vm_page_find_contig_domain() at vm_page_find_contig_domain+0xc4/frame 0xfffffe0084ffbbd0
vm_page_alloc_contig_domain() at vm_page_alloc_contig_domain+0x145/frame 0xfffffe0084ffbc60
vm_page_alloc_contig() at vm_page_alloc_contig+0x73/frame 0xfffffe0084ffbcf0
shm_dotruncate_cookie() at shm_dotruncate_cookie+0x195/frame 0xfffffe0084ffbd80
shm_truncate() at shm_truncate+0x5e/frame 0xfffffe0084ffbdc0
kern_ftruncate() at kern_ftruncate+0xb3/frame 0xfffffe0084ffbe00
amd64_syscall() at amd64_syscall+0x6a6/frame 0xfffffe0084ffbf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0084ffbf30
--- syscall (480, FreeBSD ELF64, sys_ftruncate), rip = 0x2c93c1ba985a, rsp = 0x2c93bf0e5b48, rbp = 0x2c93bf0e5b60 ---
db:0:kdb.enter.panic> dump

Dump failed. Partition too small (about 1456MB were needed this time).
Cannot dump: unknown error (error=7).
db:0:kdb.enter.panic> reset
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-20419' --destroy
+ sh -ex freebsd-ci/scripts/test/
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./
x ./auto-shutdown
x ./
x ./
x ./
x ./
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/
+ ./freebsd-ci/artifact/
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'b7ff445ffa38282daeab36ce82681ba3f54c8851', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../b7ff445ffa38282daeab36ce82681ba3f54c8851/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address