FreeBSD-main-amd64-test - Build #29557 - Failure

Reply: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #29558 - Still unstable"
In reply to: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #29556 - Still Unstable"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: <jenkins-admin_at_FreeBSD.org>
Date: Fri, 25 Sep 2026 23:07:39 UTC
FreeBSD-main-amd64-test - Build #29557 (754d2c69f3589cdd0bc4649d9e85ecd59935d9d3) - Failure

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29557/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29557/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/29557/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

4672ad1f16092c146261b4de485ec5eb5e8d127b by olce:
SYSCTL(9): Retire SYSCTL_NODE_CHILDREN()

406d6658ff821fc43eb0b2b10df4ab09eb2e2e16 by olce:
sched: options: Regroup scheduler-related kernel options

7ce0ede913801c0e74f6d90beee2fc4eb5c6ef2f by olce:
i386 NOTES: Remove IPI_PREEMPTION remnant after move to x86 NOTES

61ce73643f22f6f8273701eb978faa246bbca27d by olce:
NOTES: SMP: Move PREEMPTION out of the debugging options section

d972d832f580c307cdfd87d563a382f40877dd10 by olce:
NOTES: IPI_PREEMPTION: Fix documentation, applies to all architectures

754d2c69f3589cdd0bc4649d9e85ecd59935d9d3 by olce:
NOTES: Multiple schedulers can be compiled in at once



The end of the build log:

[...truncated 5.02 MiB...]
lock order "inpcb_wild_bucket"(sleep mutex) -> "tcphash"(sleep mutex) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80e57de5 at in_pcbinshash+0x195
#3 0xffffffff80e58717 at in_pcbconnect+0x2e7
#4 0xffffffff80e9e3ae at tcp_connect+0x7e
#5 0xffffffff80e9a47d at tcp_usr_connect+0xed
#6 0xffffffff80d157d6 at soconnectat+0xc6
#7 0xffffffff80d1debc at kern_connectat+0xec
#8 0xffffffff80d1dda3 at sys_connect+0x73
#9 0xffffffff811f2bee at amd64_syscall+0x45e
#10 0xffffffff811be79b at fast_syscall_common+0xf8

lock order "inpcb_wild_bucket"(sleep mutex) -> "inpcb_port_bucket"(sleep mutex) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80e57238 at in_pcblookup_local+0x1c8
#3 0xffffffff80e56d7c at in_pcb_lport_dest+0x37c
#4 0xffffffff80e57945 at in_pcbbind_setup_locked+0x385
#5 0xffffffff80e57a84 at in_pcbbind+0xf4
#6 0xffffffff80e9ac1e at tcp_usr_bind+0xae
#7 0xffffffff80d14681 at sobind+0x41
#8 0xffffffff80d1d436 at kern_bindat+0xd6
#9 0xffffffff80d1d2b3 at sys_bind+0x73
#10 0xffffffff811f2bee at amd64_syscall+0x45e
#11 0xffffffff811be79b at fast_syscall_common+0xf8

lock order "inpcb_port_bucket"(sleep mutex) -> "tcphash"(sleep mutex) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80e57de5 at in_pcbinshash+0x195
#3 0xffffffff80e57b2b at in_pcbbind+0x19b
#4 0xffffffff80e9ac1e at tcp_usr_bind+0xae
#5 0xffffffff80d14681 at sobind+0x41
#6 0xffffffff80d1d436 at kern_bindat+0xd6
#7 0xffffffff80d1d2b3 at sys_bind+0x73
#8 0xffffffff811f2bee at amd64_syscall+0x45e
#9 0xffffffff811be79b at fast_syscall_common+0xf8

hardcoded lock order "tcphash"(sleep mutex) -> "in6_ifaddr_lock"(rm) first seen at:
(No stack trace)

lock order "in6_ifaddr_lock"(rm) -> "lle"(rw) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c4cc0d at __rw_rlock_int+0x7d
#2 0xffffffff80eb0dbb at in6_lltable_lookup+0x10b
#3 0xffffffff80ed0f7c at nd6_lookup+0x8c
#4 0xffffffff80edcbfe at find_pfxlist_reachable_router+0x7e
#5 0xffffffff80edc9e6 at pfxlist_onlink_check+0x446
#6 0xffffffff80eda1aa at nd6_ra_input+0x177a
#7 0xffffffff80ea7e83 at icmp6_input+0x5b3
#8 0xffffffff80ec3a81 at ip6_input+0xbc1
#9 0xffffffff80dd26df at netisr_dispatch_src+0xbf
#10 0xffffffff80dafdea at ether_demux+0x17a
#11 0xffffffff80db140a at ether_nh_input+0x3ea
#12 0xffffffff80dd26df at netisr_dispatch_src+0xbf
#13 0xffffffff80db0259 at ether_input+0xe9
#14 0xffffffff82b02b94 at epair_tx_start_deferred+0xd4
#15 0xffffffff80cc24ab at taskqueue_run_locked+0x1bb
#16 0xffffffff80cc34e3 at taskqueue_thread_loop+0xd3
#17 0xffffffff80bfe5c2 at fork_exit+0x82

lock order "mld_mtx"(sleep mutex) -> "lltable"(sleep mutex) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80e41f8d at arpresolve+0x34d
#3 0xffffffff80daf630 at ether_output+0x300
#4 0xffffffff80e65f9a at ip_output+0x158a
#5 0xffffffff80db2a0a at gif_transmit+0x2fa
#6 0xffffffff80ec7e4b at ip6_output+0x1b1b
#7 0xffffffff80ece210 at mld_dispatch_packet+0x330
#8 0xffffffff80eced10 at mld_fasttimo+0x4f0
#9 0xffffffff80c71745 at softclock_call_cc+0x1b5
#10 0xffffffff80c72f26 at softclock_thread+0xc6
#11 0xffffffff80bfe5c2 at fork_exit+0x82
#12 0xffffffff811beece at fork_trampoline+0xe

lock order "if_ovpn_lock"(rm) -> "lltable"(sleep mutex) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80e41f8d at arpresolve+0x34d
#3 0xffffffff80daf630 at ether_output+0x300
#4 0xffffffff80e65f9a at ip_output+0x158a
#5 0xffffffff82b2e007 at ovpn_encap+0x6d7
#6 0xffffffff82b2e1b2 at ovpn_encrypt_tx_cb+0x142
#7 0xffffffff80f9ce18 at crypto_done+0x168
#8 0xffffffff8109661a at aesni_process+0xea
#9 0xffffffff80f9d675 at crypto_invoke+0x85
#10 0xffffffff80f9c269 at crypto_dispatch_one+0xe9
#11 0xffffffff82b2d7ca at ovpn_transmit_to_peer+0x66a
#12 0xffffffff82b2cef4 at ovpn_send_ping+0x124
#13 0xffffffff80c71745 at softclock_call_cc+0x1b5
#14 0xffffffff80c72f26 at softclock_thread+0xc6
#15 0xffffffff80bfe5c2 at fork_exit+0x82
#16 0xffffffff811beece at fork_trampoline+0xe

lock order "lltable"(sleep mutex) -> "lle"(rw) first seen at:
#0 0xffffffff80ccff3a at witness_checkorder+0x36a
#1 0xffffffff80c4be85 at _rw_wlock_cookie+0x65
#2 0xffffffff80ed32ac at nd6_add_ifa_lle+0xcc
#3 0xffffffff80eae580 at in6_update_ifa+0xdb0
#4 0xffffffff80ead08f at in6_addifaddr+0x7f
#5 0xffffffff80eacea1 at in6_control_ioctl+0x9b1
#6 0xffffffff80f05b1e at rtnl_handle_addr+0x62e
#7 0xffffffff80f01770 at rtnl_handle_message+0x1a0
#8 0xffffffff80efff3a at nl_receive_message+0x12a
#9 0xffffffff80eff875 at nl_taskqueue_handler+0x3f5
#10 0xffffffff80cc24ab at taskqueue_run_locked+0x1bb
#11 0xffffffff80cc34e3 at taskqueue_thread_loop+0xd3
#12 0xffffffff80bfe5c2 at fork_exit+0x82
#13 0xffffffff811beece at fork_trampoline+0xe

lock order lle -> in6_multi_list_mtx attempted at:
#0 0xffffffff80cd090b at witness_checkorder+0xd3b
#1 0xffffffff80c28c61 at __mtx_lock_flags+0x91
#2 0xffffffff80eb1201 at in6_lltable_delete_entry+0xc1
#3 0xffffffff80db431a at lltable_delete_addr+0xda
#4 0xffffffff80f067f7 at rtnl_handle_delneigh+0x97
#5 0xffffffff80f0170d at rtnl_handle_message+0x13d
#6 0xffffffff80efff3a at nl_receive_message+0x12a
#7 0xffffffff80eff875 at nl_taskqueue_handler+0x3f5
#8 0xffffffff80cc24ab at taskqueue_run_locked+0x1bb
#9 0xffffffff80cc34e3 at taskqueue_thread_loop+0xd3
#10 0xffffffff80bfe5c2 at fork_exit+0x82
#11 0xffffffff811beece at fork_trampoline+0xe
lock order reversal: (sleepable after non-sleepable)
 1st 0xfffff801289bb290 lle (lle, rw) @ /usr/src/sys/netinet6/in6.c:2461
 2nd 0xffffffff81ea2d80 in6_multi_sx (in6_multi_sx, sx) @ /usr/src/sys/netinet6/in6_mcast.c:1334
lock order lle -> in6_multi_sx attempted at:
#0 0xffffffff80cd090b at witness_checkorder+0xd3b
#1 0xffffffff80c5cc00 at _sx_xlock+0x60
#2 0xffffffff80eb7ba7 at in6_leavegroup+0x27
#3 0xffffffff80eb124c at in6_lltable_delete_entry+0x10c
#4 0xffffffff80db431a at lltable_delete_addr+0xda
#5 0xffffffff80f067f7 at rtnl_handle_delneigh+0x97
#6 0xffffffff80f0170d at rtnl_handle_message+0x13d
#7 0xffffffff80efff3a at nl_receive_message+0x12a
#8 0xffffffff80eff875 at nl_taskqueue_handler+0x3f5
#9 0xffffffff80cc24ab at taskqueue_run_locked+0x1bb
#10 0xffffffff80cc34e3 at taskqueue_thread_loop+0xd3
#11 0xffffffff80bfe5c2 at fork_exit+0x82
#12 0xffffffff811beece at fork_trampoline+0xe
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
passed  [5.020s]
sys/netinet6/proxy_ndp:pndp_ifdestroy_success  ->  epair0a: Ethernet address: 26:2d:01:d1:c3:e1
epair0b: Ethernet address: ea:56:62:7f:7f:f8
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
passed  [2.022s]
sys/netinet6/proxy_ndp:pndp_neighbor_advert  ->  epair0a: Ethernet address: ba:2e:31:45:40:b5
epair0b: Ethernet address: b6:ff:18:51:8e:a8
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
passed  [4.128s]
sys/netinet6/redirect:valid_redirect  ->  epair0a: Ethernet address: 8a:59:48:62:a5:1a
epair0b: Ethernet address: 76:75:e6:56:d8:c7
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: promiscuous mode enabled
epair0a: promiscuous mode disabled
epair0b: link state changed to DOWN
epair0a: link state changed to DOWN
passed  [3.319s]
sys/netinet6/scapyi386:scapyi386  ->  epair0a: Ethernet address: 66:6c:f0:3d:f7:8a
epair0b: Ethernet address: 8a:67:0b:2b:78:bb
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: promiscuous mode enabled
epair0a: promiscuous mode disabled
epair0b: link state changed to DOWN
epair0a: link state changed to DOWN
passed  [4.588s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_base  ->  broken: Test case body timed out  [300.007s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_nhop  ->  broken: Test case body timed out  [300.004s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_pktinfo[empty]  ->  broken: Test case body timed out  [300.010s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_pktinfo[ifsame]  ->  broken: Test case body timed out  [300.007s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_pktinfo[ipandif]  ->  broken: Test case body timed out  [300.010s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_pktinfo[iponly1]  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6Output::test_output6_pktinfo[nolocalip]  ->  broken: Test case body timed out  [300.012s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLL::test_output6_linklocal  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[gu-no_sav]  ->  broken: Test case body timed out  [300.013s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[gu-sav]  ->  broken: Test case body timed out  [300.010s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[ll-no_sav]  ->  broken: Test case body timed out  [300.008s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[ll-sav]  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[lo-no_sav]  ->  broken: Test case body timed out  [300.011s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_tcp[lo-sav]  ->  broken: Test case body timed out  [300.014s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[gu-no_sav]  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[gu-sav]  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[ll-no_sav]  ->  broken: Test case body timed out  [300.013s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[ll-sav]  ->  broken: Test case body timed out  [300.007s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[lo-no_sav]  ->  broken: Test case body timed out  [300.009s]
sys/netinet6/test_ip6_output.py:TestIP6OutputLoopback::test_output6_self_udp[lo-sav]  ->  broken: Test case body timed out  [300.005s]
sys/netinet6/test_ip6_output.py:TestIP6OutputMulticast::test_output6_multicast[ff02]  ->  broken: Test case body timed out  [300.004s]
sys/netinet6/test_ip6_output.py:TestIP6OutputMulticast::test_output6_multicast[ff05]  ->  2026-09-25T23:06:37.585185+00:00 - auditd 1426 - - auditd_wait_for_events: SIGTERM
2026-09-25T23:06:37.587483+00:00 - auditd 1426 - - Auditing disabled
2026-09-25T23:06:37.598624+00:00 - auditd 1426 - - renamed /var/audit/20260925170943.not_terminated to /var/audit/20260925170943.20260925230637
2026-09-25T23:06:37.600757+00:00 - auditd 1426 - - Finished
2026-09-25T23:06:57.593791+00:00 - init 1 - - some processes would not die; ps axl advised
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-29557' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./disable-notyet-tests.sh
x ./run.sh
x ./auto-shutdown
x ./disable-zfs-tests.sh
x ./run-kyua.sh
x ./disable-dtrace-tests.sh
+ rm -f test-report.txt test-report.xml
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins11648755109527935359.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '754d2c69f3589cdd0bc4649d9e85ecd59935d9d3', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../754d2c69f3589cdd0bc4649d9e85ecd59935d9d3/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address