FreeBSD-main-amd64-test - Build #19786 - Still Failing

From: <>
Date: Fri, 12 Nov 2021 17:56:14 UTC
FreeBSD-main-amd64-test - Build #19786 (eb6b6622fe858fef7896bb25a8a79dbd08b01335) - Still Failing

Build information:
Full change log:
Full build log:

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

0359e7a5e49f3fa8f9e87923a0830239d3372132 by mjg:
net: sprinkle __predict_false in ip_input on error conditions

eb6b6622fe858fef7896bb25a8a79dbd08b01335 by imp:
Update MINIMAL to have CC options

The end of the build log:

[...truncated 3.93 MB...]
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2134dc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2134de0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2134e20
_sleep() at _sleep+0x29e/frame 0xfffffe00b2134ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b2134ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2134f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2134f30
--- trap 0x5a5a5a5a, rip = 0x5a5a5a5a5a5a5a5a, rsp = 0x5a5a5a5a5a5a5a5a, rbp = 0x5a5a5a5a5a5a5a5a ---

Tracing command zfskern pid 19217 tid 100153 td 0xfffffe00b35bd3a0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b212fc20
mi_switch() at mi_switch+0x155/frame 0xfffffe00b212fc40
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b212fc80
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b212fcc0
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x19b/frame 0xfffffe00b212fd40
l2arc_feed_thread() at l2arc_feed_thread+0x274/frame 0xfffffe00b212fef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b212ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b212ff30
--- trap 0x5a5a5a5a, rip = 0x5a5a5a5a5a5a5a5a, rsp = 0x5a5a5a5a5a5a5a5a, rbp = 0x5a5a5a5a5a5a5a5a ---

Tracing command zfskern pid 19217 tid 100154 td 0xfffffe00b35bcc80
sched_switch() at sched_switch+0x578/frame 0xfffffe00b212adc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b212ade0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b212ae20
_sleep() at _sleep+0x29e/frame 0xfffffe00b212aec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00b212aef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b212af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b212af30
--- trap 0x5a5a5a5a, rip = 0x5a5a5a5a5a5a5a5a, rsp = 0x5a5a5a5a5a5a5a5a, rbp = 0x5a5a5a5a5a5a5a5a ---

Tracing command zfskern pid 19217 tid 100155 td 0xfffffe00b35bc560
sched_switch() at sched_switch+0x578/frame 0xfffffe00b2125cc0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b2125ce0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b2125d20
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00b2125d70
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00b2125d80
_cv_wait_sig() at _cv_wait_sig+0x165/frame 0xfffffe00b2125de0
zfs_zevent_wait() at zfs_zevent_wait+0x47/frame 0xfffffe00b2125e00
sysevent_worker() at sysevent_worker+0x49/frame 0xfffffe00b2125ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b2125f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b2125f30
--- trap 0x5a5a5a5a, rip = 0x5a5a5a5a5a5a5a5a, rsp = 0x5a5a5a5a5a5a5a5a, rbp = 0x5a5a5a5a5a5a5a5a ---

Tracing command aiod2 pid 19226 tid 100102 td 0xfffffe00b2d43900
sched_switch() at sched_switch+0x578/frame 0xfffffe0063be0d60
mi_switch() at mi_switch+0x155/frame 0xfffffe0063be0d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063be0dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe0063be0e00
_sleep() at _sleep+0x282/frame 0xfffffe0063be0ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe0063be0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe0063be0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0063be0f30
--- trap 0xc, rip = 0x8011a8bba, rsp = 0x7fffffffeae8, rbp = 0x7fffffffeb00 ---

Tracing command ng_queue pid 9145 tid 100589 td 0xfffffe00b35be900
sched_switch() at sched_switch+0x578/frame 0xfffffe00b21d9d80
mi_switch() at mi_switch+0x155/frame 0xfffffe00b21d9da0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b21d9de0
_sleep() at _sleep+0x29e/frame 0xfffffe00b21d9e80
ngthread() at ngthread+0x88/frame 0xfffffe00b21d9ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b21d9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b21d9f30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe868, rbp = 0x7fffffffe960 ---

Tracing command ng_queue pid 9145 tid 101576 td 0xfffffe00b4ecc740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b5031d80
mi_switch() at mi_switch+0x155/frame 0xfffffe00b5031da0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b5031de0
_sleep() at _sleep+0x29e/frame 0xfffffe00b5031e80
ngthread() at ngthread+0x88/frame 0xfffffe00b5031ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b5031f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b5031f30
--- trap 0xc, rip = 0x8011f2bba, rsp = 0x7fffdfdfcf48, rbp = 0x7fffdfdfcf60 ---

Tracing command aiod19 pid 58751 tid 100670 td 0xfffffe00b4e7a1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3ddad60
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3ddad80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3ddadc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3ddae00
_sleep() at _sleep+0x282/frame 0xfffffe00b3ddaea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00b3ddaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3ddaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3ddaf30
--- trap 0xc, rip = 0x8011a7bba, rsp = 0x7fffffffea08, rbp = 0x7fffffffea20 ---

Tracing command aiod27 pid 58759 tid 100654 td 0xfffffe00b4e39740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3d7bd60
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3d7bd80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3d7bdc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3d7be00
_sleep() at _sleep+0x282/frame 0xfffffe00b3d7bea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00b3d7bef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3d7bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3d7bf30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffe9d0 ---

Tracing command aiod32 pid 58764 tid 100593 td 0xfffffe00b4dc1560
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3cd6d60
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3cd6d80
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3cd6dc0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3cd6e00
_sleep() at _sleep+0x282/frame 0xfffffe00b3cd6ea0
aio_daemon() at aio_daemon+0x305/frame 0xfffffe00b3cd6ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3cd6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3cd6f30
--- trap 0xc, rip = 0x8011e0bba, rsp = 0x7fffffffea78, rbp = 0x7fffffffea90 ---

Tracing command soaiod1 pid 58771 tid 100628 td 0xfffffe00b4dc4740
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3c27d70
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3c27d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3c27dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3c27e10
_sleep() at _sleep+0x282/frame 0xfffffe00b3c27eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00b3c27ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3c27f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3c27f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command soaiod2 pid 58772 tid 100602 td 0xfffffe00b35be1e0
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3ce0d70
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3ce0d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3ce0dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3ce0e10
_sleep() at _sleep+0x282/frame 0xfffffe00b3ce0eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00b3ce0ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3ce0f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3ce0f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command soaiod3 pid 58773 tid 100575 td 0xfffffe00b3615900
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3cead70
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3cead90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3ceadd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3ceae10
_sleep() at _sleep+0x282/frame 0xfffffe00b3ceaeb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00b3ceaef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3ceaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3ceaf30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd538, rbp = 0x7fffffffd550 ---

Tracing command soaiod4 pid 58774 tid 100675 td 0xfffffe00b4e82720
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3d03d70
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3d03d90
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3d03dd0
sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00b3d03e10
_sleep() at _sleep+0x282/frame 0xfffffe00b3d03eb0
soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00b3d03ef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3d03f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3d03f30
--- trap 0xc, rip = 0x8011edbba, rsp = 0x7fffffffd528, rbp = 0x7fffffffd540 ---

Tracing command g_mp_kt pid 1476 tid 100590 td 0xfffffe00b35bf020
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3cdbda0
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3cdbdc0
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3cdbe00
_sleep() at _sleep+0x29e/frame 0xfffffe00b3cdbea0
g_multipath_kt() at g_multipath_kt+0x265/frame 0xfffffe00b3cdbef0
fork_exit() at fork_exit+0x80/frame 0xfffffe00b3cdbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00b3cdbf30
--- trap 0xc, rip = 0x801209bba, rsp = 0x7fffffffe848, rbp = 0x7fffffffe940 ---

Tracing command sh pid 12063 tid 100608 td 0xfffffe00b35f9560
sched_switch() at sched_switch+0x578/frame 0xfffffe00b3c45a10
mi_switch() at mi_switch+0x155/frame 0xfffffe00b3c45a30
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe00b3c45a70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe00b3c45ac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00b3c45ad0
_sleep() at _sleep+0x294/frame 0xfffffe00b3c45b70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe00b3c45c00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe00b3c45e00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe00b3c45f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b3c45f30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80120a15a, rsp = 0x7fffffffdae8, rbp = 0x7fffffffdb90 ---

Tracing command atf-check pid 12115 tid 100580 td 0xfffffe00b361cc80
sched_switch() at sched_switch+0x578/frame 0xfffffe0063baea10
mi_switch() at mi_switch+0x155/frame 0xfffffe0063baea30
sleepq_switch() at sleepq_switch+0x119/frame 0xfffffe0063baea70
sleepq_catch_signals() at sleepq_catch_signals+0x266/frame 0xfffffe0063baeac0
sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe0063baead0
_sleep() at _sleep+0x294/frame 0xfffffe0063baeb70
kern_wait6() at kern_wait6+0x474/frame 0xfffffe0063baec00
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0063baee00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe0063baef30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063baef30
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80134215a, rsp = 0x7fffffffe648, rbp = 0x7fffffffe680 ---

Tracing command python3.8 pid 12116 tid 101945 td 0xfffffe00b3fc61e0 (CPU 0)
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00b8536990
vpanic() at vpanic+0x1b8/frame 0xfffffe00b85369f0
panic() at panic+0x43/frame 0xfffffe00b8536a50
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00b8536ab0
trap_pfault() at trap_pfault+0x99/frame 0xfffffe00b8536b10
trap() at trap+0x2a7/frame 0xfffffe00b8536c20
calltrap() at calltrap+0x8/frame 0xfffffe00b8536c20
--- trap 0xc, rip = 0xffffffff80e1dd47, rsp = 0xfffffe00b8536cf0, rbp = 0xfffffe00b8536d20 ---
tcp_newtcpcb() at tcp_newtcpcb+0x127/frame 0xfffffe00b8536d20
tcp_usr_attach() at tcp_usr_attach+0x103/frame 0xfffffe00b8536d50
socreate() at socreate+0x1cf/frame 0xfffffe00b8536da0
kern_socket() at kern_socket+0xb8/frame 0xfffffe00b8536e00
amd64_syscall() at amd64_syscall+0x5cd/frame 0xfffffe00b8536f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b8536f30
--- syscall (97, FreeBSD ELF64, sys_socket), rip = 0x80076a47a, rsp = 0x7fffffff5ba8, rbp = 0x7fffffff5c80 ---
db:0:kdb.enter.panic> dump
Dumping 438 out of 4058 MB:..4%..11%..22%..33%..41%..52%..63%..73%..81%..92%
Dump complete
db:0:kdb.enter.panic> reset
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-19786' --destroy
+ sh -ex freebsd-ci/scripts/test/
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./
x ./auto-shutdown
x ./
x ./
x ./
x ./
+ rm -f 'test-report.*'
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/
+ ./freebsd-ci/artifact/
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': 'eb6b6622fe858fef7896bb25a8a79dbd08b01335', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../eb6b6622fe858fef7896bb25a8a79dbd08b01335/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address