FreeBSD-main-amd64-test - Build #29895 - Failure

Reply: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #29896 - Still unstable"
In reply to: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #29894 - Still unstable"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: <jenkins-admin_at_FreeBSD.org>
Date: Sat, 07 Nov 2026 18:15:08 UTC
FreeBSD-main-amd64-test - Build #29895 (4ca929100407f62aba146aa04bb0c8c44d1fe292) - Failure

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29895/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29895/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29895/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

7b18863d597e0dad0734cf6e06c5e7689be2047c by brooks:
mac syscalls: add kern_ implementations

d7c0ab90cde19e2184f2d76d0fe43e85c74785d4 by brooks:
acct(2): add kern_acct

3369af6f98ca606877e2c6c96e67f082d3f0f25d by brooks:
cpuset(2): add kern_cpuset

230d581d4e8c3017d4e711dc97f2b53ebe1ea7ee by brooks:
fstat(2): add user_fstat

4ca929100407f62aba146aa04bb0c8c44d1fe292 by brooks:
kenv(2): add kern_kenv



The end of the build log:

[...truncated 6.58 MiB...]
#5 0xffffffff80da6f9d at ifioctl+0x73d
#6 0xffffffff80cd565d at kern_ioctl+0x2bd
#7 0xffffffff80cd533f at sys_ioctl+0x12f
#8 0xffffffff811fac9e at amd64_syscall+0x45e
#9 0xffffffff811c679b at fast_syscall_common+0xf8
lock order reversal:
 1st 0xfffff8012ff68f10 pfsync (pfsync, sleep mutex) @ /usr/src/sys/netpfil/pf/if_pfsync.c:3148
 2nd 0xffffffff81ea1db8 in6_ifaddr_lock (in6_ifaddr_lock, rm) @ /usr/src/sys/netinet6/in6_src.c:292
All lock orders from in6_ifaddr_lock -> pfsync:
"in6_ifaddr_lock" -> "nd6 DAD queue" -> "if_addr_lock" -> "pf rulesets" -> "pf_idhash" -> "pfsync"
"in6_ifaddr_lock" -> "nd6 DAD queue" -> "if_addr_lock" -> "pf rulesets" -> "pf_keyhash" -> "pf_idhash" -> "pfsync"
"in6_ifaddr_lock" -> "nd6 DAD queue" -> "if_addr_lock" -> "pf rulesets" -> "pfsync bulk" -> "pf_idhash" -> "pfsync"

lock order "in6_ifaddr_lock"(rm) -> "nd6 DAD queue"(rw) first seen at:
(No stack trace)

lock order "nd6 DAD queue"(rw) -> "if_addr_lock"(sleep mutex) first seen at:
#0 0xffffffff80ccd82a at witness_checkorder+0x36a
#1 0xffffffff80c2e481 at __mtx_lock_flags+0x91
#2 0xffffffff80ed3f6d at nd6_queue_add+0x7d
#3 0xffffffff80ed5dbc at nd6_grand_start+0x12c
#4 0xffffffff80ed644d at nd6_dad_timer+0x52d
#5 0xffffffff80c77038 at softclock_call_cc+0x1c8
#6 0xffffffff80c78846 at softclock_thread+0xc6
#7 0xffffffff80c03c72 at fork_exit+0x82
#8 0xffffffff811c6ece at fork_trampoline+0xe

lock order "if_addr_lock"(sleep mutex) -> "pf rulesets"(rm) first seen at:
(No stack trace)

lock order "pf rulesets"(rm) -> "pf_idhash"(sleep mutex) first seen at:
#0 0xffffffff80ccd82a at witness_checkorder+0x36a
#1 0xffffffff80c2e481 at __mtx_lock_flags+0x91
#2 0xffffffff82c93e76 at pf_clear_all_states+0xa6
#3 0xffffffff82c93aa5 at vnet_pf_uninit+0xc65
#4 0xffffffff80dedf14 at vnet_destroy+0x154
#5 0xffffffff80c0e736 at prison_deref+0xaf6
#6 0xffffffff80c0fe57 at sys_jail_remove+0x1a7
#7 0xffffffff811fac9e at amd64_syscall+0x45e
#8 0xffffffff811c679b at fast_syscall_common+0xf8

lock order "pf rulesets"(rm) -> "pf_keyhash"(sleep mutex) first seen at:
#0 0xffffffff80ccd82a at witness_checkorder+0x36a
#1 0xffffffff80c2e481 at __mtx_lock_flags+0x91
#2 0xffffffff82c6b052 at pf_find_state+0xb2
#3 0xffffffff82c731f5 at pf_test_state+0x135
#4 0xffffffff82c6f3ed at pf_test+0x287d
#5 0xffffffff82c92231 at pf_check_out+0x41
#6 0xffffffff80dd46a8 at pfil_mbuf_out+0x58
#7 0xffffffff80e63ae0 at ip_output+0xe70
#8 0xffffffff80e9e6d1 at udp_send+0xb51
#9 0xffffffff80d136b5 at sosend_dgram+0x315
#10 0xffffffff80d142a9 at sousrsend+0x79
#11 0xffffffff80d1c0fe at kern_sendit+0x1ce
#12 0xffffffff80d1c3fd at sendit+0x18d
#13 0xffffffff80d1c258 at sys_sendto+0x58
#14 0xffffffff811fac9e at amd64_syscall+0x45e
#15 0xffffffff811c679b at fast_syscall_common+0xf8

lock order "pf_keyhash"(sleep mutex) -> "pf_idhash"(sleep mutex) first seen at:
#0 0xffffffff80ccd82a at witness_checkorder+0x36a
#1 0xffffffff80c2e481 at __mtx_lock_flags+0x91
#2 0xffffffff82c61874 at pf_state_insert+0x954
#3 0xffffffff82c72505 at pf_test_rule+0x20c5
#4 0xffffffff82c6f44d at pf_test+0x28dd
#5 0xffffffff82c92231 at pf_check_out+0x41
#6 0xffffffff80dd46a8 at pfil_mbuf_out+0x58
#7 0xffffffff80e63ae0 at ip_output+0xe70
#8 0xffffffff80e9e6d1 at udp_send+0xb51
#9 0xffffffff80d136b5 at sosend_dgram+0x315
#10 0xffffffff80d142a9 at sousrsend+0x79
#11 0xffffffff80d1c0fe at kern_sendit+0x1ce
#12 0xffffffff80d1c3fd at sendit+0x18d
#13 0xffffffff80d1c258 at sys_sendto+0x58
#14 0xffffffff811fac9e at amd64_syscall+0x45e
#15 0xffffffff811c679b at fast_syscall_common+0xf8

lock order "pf rulesets"(rm) -> "pfsync bulk"(sleep mutex) first seen at:
(No stack trace)

lock order "pfsync bulk"(sleep mutex) -> "pf_idhash"(sleep mutex) first seen at:
(No stack trace)

lock order "pf_idhash"(sleep mutex) -> "pfsync"(sleep mutex) first seen at:
#0 0xffffffff80ccd82a at witness_checkorder+0x36a
#1 0xffffffff80c2e481 at __mtx_lock_flags+0x91
#2 0xffffffff82ce04d5 at pfsync_defer+0xc5
#3 0xffffffff82c70f2d at pf_test_rule+0xaed
#4 0xffffffff82c6f44d at pf_test+0x28dd
#5 0xffffffff82c92231 at pf_check_out+0x41
#6 0xffffffff80dd46a8 at pfil_mbuf_out+0x58
#7 0xffffffff80e63ae0 at ip_output+0xe70
#8 0xffffffff80e9e6d1 at udp_send+0xb51
#9 0xffffffff80d136b5 at sosend_dgram+0x315
#10 0xffffffff80d142a9 at sousrsend+0x79
#11 0xffffffff80d1c0fe at kern_sendit+0x1ce
#12 0xffffffff80d1c3fd at sendit+0x18d
#13 0xffffffff80d1c258 at sys_sendto+0x58
#14 0xffffffff811fac9e at amd64_syscall+0x45e
#15 0xffffffff811c679b at fast_syscall_common+0xf8

lock order pfsync -> in6_ifaddr_lock attempted at:
#0 0xffffffff80cce1fb at witness_checkorder+0xd3b
#1 0xffffffff80c502d9 at _rm_rlock_debug+0x129
#2 0xffffffff80ebda2e at in6_selectsrc+0x3ee
#3 0xffffffff80ebea41 at in6_selectsrc_addr+0x71
#4 0xffffffff82cdfddd at pfsync_kstatus_to_softc+0x51d
#5 0xffffffff82cdf1ab at pfsyncioctl+0x78b
#6 0xffffffff80da6f9d at ifioctl+0x73d
#7 0xffffffff80cd565d at kern_ioctl+0x2bd
#8 0xffffffff80cd533f at sys_ioctl+0x12f
#9 0xffffffff811fac9e at amd64_syscall+0x45e
#10 0xffffffff811c679b at fast_syscall_common+0xf8
carp: demoted by 240 to 240 (pfsync bulk start)
epair1a: link state changed to DOWN
epair1b: link state changed to DOWN
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (pfsync destroy)
epair2a: link state changed to DOWN
epair2b: link state changed to DOWN
carp: demoted by -240 to 0 (pfsync destroy)
passed  [4.543s]
sys/netpfil/pf/pfsync:basic_ipv6_unicast  ->  epair0a: Ethernet address: 06:d3:95:22:86:2f
epair0b: Ethernet address: 6a:b9:67:95:ef:d9
epair0a: link state changed to UP
epair0b: link state changed to UP
epair1a: Ethernet address: ce:6c:a0:b4:50:87
epair1b: Ethernet address: fa:63:50:b1:be:61
epair1a: link state changed to UP
epair1b: link state changed to UP
epair2a: Ethernet address: 56:3b:28:59:86:39
epair2b: Ethernet address: e2:a9:45:45:36:10
epair2a: link state changed to UP
epair2b: link state changed to UP
carp: demoted by 240 to 240 (pfsync bulk start)
carp: demoted by 240 to 240 (pfsync bulk start)
epair1a: link state changed to DOWN
epair1b: link state changed to DOWN
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (pfsync destroy)
epair2a: link state changed to DOWN
epair2b: link state changed to DOWN
carp: demoted by -240 to 0 (pfsync destroy)
passed  [2.440s]
sys/netpfil/pf/pfsync:bulk  ->  epair0a: Ethernet address: 06:a0:65:cf:74:53
epair0b: Ethernet address: 2e:1c:45:42:70:56
epair0a: link state changed to UP
epair0b: link state changed to UP
epair1a: Ethernet address: ee:81:36:62:d9:30
epair1b: Ethernet address: 6a:8b:6a:a8:a5:06
epair1a: link state changed to UP
epair1b: link state changed to UP
epair2a: Ethernet address: b6:f8:d7:ad:93:2b
epair2b: Ethernet address: 32:c0:3c:93:ba:1d
epair2a: link state changed to UP
epair2b: link state changed to UP
carp: demoted by 240 to 240 (pfsync bulk start)
epair1b: promiscuous mode enabled
epair1b: promiscuous mode disabled
carp: demoted by 240 to 240 (pfsync bulk start)
carp: demoted by -240 to 0 (pfsync bulk done)
epair1a: link state changed to DOWN
epair1b: link state changed to DOWN
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (pfsync destroy)
epair2a: link state changed to DOWN
epair2b: link state changed to DOWN
broken: Test case cleanup timed out  [5.072s]
sys/netpfil/pf/pfsync:defer  ->  panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffff8010241a780 (swi6: task queue), blocked for 180184 ticks

cpuid = 0
time = 1794075307
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe008bdded20
vpanic() at vpanic+0x149/frame 0xfffffe008bddee50
panic() at panic+0x43/frame 0xfffffe008bddeeb0
deadlkres() at deadlkres+0x32a/frame 0xfffffe008bddeef0
fork_exit() at fork_exit+0x82/frame 0xfffffe008bddef30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008bddef30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 0 tid 100079 ]
Stopped at      kdb_enter+0x33: movq    $0,0x115fa72(%rip)
db:0:kdb.enter.panic> show pcpu
cpuid        = 0
dynamic pcpu = 0x9b4380
curthread    = 0xfffff80102452000: pid 0 tid 100079 critnest 1 "deadlkres"
curpcb       = 0xfffff80102452550
fpcurthread  = none
idlethread   = 0xfffff8010240b780: tid 100003 "idle: cpu0"
self         = 0xffffffff82610000
curpmap      = 0xffffffff81da8d00
tssp         = 0xffffffff82610384
rsp0         = 0xfffffe008bddf000
kcr3         = 0x8000000002552002
ucr3         = 0xffffffffffffffff
scr3         = 0x1c91feac6
gs32p        = 0xffffffff82610404
ldt          = 0xffffffff82610444
tss          = 0xffffffff82610434
curvnet      = 0
spin locks held:
db:0:kdb.enter.panic>  reset
Uptime: 2h28m36s
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-29895' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./run.sh
x ./auto-shutdown
x ./disable-dtrace-tests.sh
x ./run-kyua.sh
x ./disable-zfs-tests.sh
x ./disable-notyet-tests.sh
+ rm -f test-report.txt test-report.xml
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins10520934052952761004.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '4ca929100407f62aba146aa04bb0c8c44d1fe292', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../4ca929100407f62aba146aa04bb0c8c44d1fe292/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address