Re: FreeBSD-main-amd64-test - Build #26985 - Still Failing

Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Rick Macklem <rick.macklem_at_gmail.com>
Date: Tue, 14 Oct 2025 13:31:15 UTC
Just fyi, I saw this to-day, rick

On Tue, Oct 14, 2025 at 12:17 AM <jenkins-admin@freebsd.org> wrote:
>
> CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
>
> FreeBSD-main-amd64-test - Build #26985 (4fc1e7546f1289183ec5d0b80653cab090c60399) - Still Failing
>
> Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/26985/
> Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/26985/changes
> Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/26985/console
>
> Status explanation:
> "Failure" - the build is suspected being broken by the following changes
> "Still Failing" - the build has not been fixed by the following changes and
>                   this is a notification to note that these changes have
>                   not been fully tested by the CI system
>
> Change summaries:
> (Those commits are likely but not certainly responsible)
>
> 4fc1e7546f1289183ec5d0b80653cab090c60399 by rmacklem:
> RELNOTES: Fix a typo in the 5000d023a446 entry
>
>
>
> The end of the build log:
>
> [...truncated 3.74 MiB...]
> epair0b: link state changed to DOWN
> epair1a: link state changed to DOWN
> epair1b: link state changed to DOWN
> passed  [0.267s]
> sys/netinet/multicast:MCAST_JOIN_GROUP  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair1a: Ethernet address: 58:9c:fc:10:56:f8
> epair1b: Ethernet address: 58:9c:fc:10:8b:12
> epair1a: link state changed to UP
> epair1b: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> epair1a: link state changed to DOWN
> epair1b: link state changed to DOWN
> passed  [0.245s]
> sys/netinet/output:output_raw_flowid_mpath_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair1a: Ethernet address: 58:9c:fc:10:56:f8
> epair1b: Ethernet address: 58:9c:fc:10:8b:12
> epair1a: link state changed to UP
> epair1b: link state changed to UP
> lo1: link state changed to UP
> lo2: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> epair1a: link state changed to DOWN
> epair1b: link state changed to DOWN
> passed  [0.620s]
> sys/netinet/output:output_raw_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> passed  [0.193s]
> sys/netinet/output:output_tcp_flowid_mpath_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair1a: Ethernet address: 58:9c:fc:10:56:f8
> epair1b: Ethernet address: 58:9c:fc:10:8b:12
> epair1a: link state changed to UP
> epair1b: link state changed to UP
> lo1: link state changed to UP
> lo2: link state changed to UP
> lock order reversal:
>  1st 0xffffffff81a01980 allproc (allproc, sx) @ /usr/src/sys/kern/kern_descrip.c:4620
>  2nd 0xfffffe000ee27510 filedesc structure (filedesc structure, sx) @ /usr/src/sys/kern/kern_descrip.c:4637
> lock order allproc -> filedesc structure attempted at:
> #0 0xffffffff80c05db1 at witness_checkorder+0xbe1
> #1 0xffffffff80b99e04 at _sx_slock_int+0x64
> #2 0xffffffff80b29fee at sysctl_kern_file+0x1fe
> #3 0xffffffff80b9ffdc at sysctl_root_handler_locked+0x9c
> #4 0xffffffff80b9f33f at sysctl_root+0x22f
> #5 0xffffffff80b9fa26 at userland_sysctl+0x196
> #6 0xffffffff80b9fdd6 at kern___sysctlbyname+0x226
> #7 0xffffffff80b9fe1d at sys___sysctlbyname+0x2d
> #8 0xffffffff810e2c71 at amd64_syscall+0x451
> #9 0xffffffff810b3e4b at fast_syscall_common+0xf8
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> epair1a: link state changed to DOWN
> epair1b: link state changed to DOWN
> passed  [2.428s]
> sys/netinet/output:output_tcp_setup_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> passed  [0.250s]
> sys/netinet/output:output_udp_flowid_mpath_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair1a: Ethernet address: 58:9c:fc:10:56:f8
> epair1b: Ethernet address: 58:9c:fc:10:8b:12
> epair1a: link state changed to UP
> epair1b: link state changed to UP
> lo1: link state changed to UP
> lo2: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> epair1a: link state changed to DOWN
> epair1b: link state changed to DOWN
> passed  [8.346s]
> sys/netinet/output:output_udp_setup_success  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair0a: link state changed to DOWN
> epair0b: link state changed to DOWN
> passed  [1.373s]
> sys/netinet/redirect:valid_redirect  ->  epair0a: Ethernet address: 58:9c:fc:10:c4:f5
> epair0b: Ethernet address: 58:9c:fc:10:64:e7
> epair0a: link state changed to UP
> epair0b: link state changed to UP
> epair0a: promiscuous mode enabled
> epair0a: promiscuous mode disabled
> epair0b: link state changed to DOWN
> epair0a: link state changed to DOWN
> passed  [1.134s]
> sys/netinet/so_reuseport_lb_test:basic_ipv4  ->  passed  [0.783s]
> sys/netinet/so_reuseport_lb_test:basic_ipv6  ->  Limiting tcp reset response from 19818 to 210 packets/sec
> passed  [0.855s]
> sys/netinet/so_reuseport_lb_test:bind_without_listen  ->  passed  [0.005s]
> sys/netinet/so_reuseport_lb_test:concurrent_add  ->  passed  [2.597s]
> sys/netinet/so_reuseport_lb_test:connect_bound  ->  passed  [0.004s]
> sys/netinet/so_reuseport_lb_test:connect_not_bound  ->  passed  [0.004s]
> sys/netinet/so_reuseport_lb_test:double_listen_ipv4  ->  passed  [0.004s]
> sys/netinet/so_reuseport_lb_test:double_listen_ipv6  ->  passed  [0.004s]
> sys/netinet/socket_afinet:socket_afinet  ->  passed  [0.003s]
> sys/netinet/socket_afinet:socket_afinet_bind_connected_port  ->  passed  [0.006s]
> sys/netinet/socket_afinet:socket_afinet_bind_ok  ->  passed  [0.003s]
> sys/netinet/socket_afinet:socket_afinet_bind_zero  ->  skipped: doesn't work when mac_portacl(4) loaded (https://bugs.freebsd.org/238781)  [0.004s]
> sys/netinet/socket_afinet:socket_afinet_bindany  ->  passed  [0.004s]
> sys/netinet/socket_afinet:socket_afinet_multibind  ->  passed  [0.120s]
> sys/netinet/socket_afinet:socket_afinet_poll_no_rdhup  ->  passed  [0.004s]
> sys/netinet/socket_afinet:socket_afinet_poll_rdhup  ->  passed  [0.004s]
> sys/netinet/socket_afinet:socket_afinet_stream_reconnect  ->  passed  [0.004s]
> sys/netinet/tcp_connect_port_test:basic_ipv4  ->  Limiting tcp reset response from 12313 to 194 packets/sec
> Limiting tcp reset response from 20725 to 198 packets/sec
> Limiting tcp reset response from 17486 to 202 packets/sec
> Limiting tcp reset response from 13841 to 216 packets/sec
> passed  [3.897s]
> sys/netinet/tcp_connect_port_test:basic_ipv6  ->  Limiting tcp reset response from 13471 to 191 packets/sec
> Limiting tcp reset response from 13243 to 215 packets/sec
> Limiting tcp reset response from 13439 to 203 packets/sec
> Limiting tcp reset response from 13479 to 188 packets/sec
> Limiting tcp reset response from 13647 to 215 packets/sec
> passed  [4.864s]
> TCP HPTS started 2 ((unbounded)) swi interrupt threads
> sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_concurrent_operations  ->  failed: /usr/tests/atf_python/sys/netlink/netlink.py:376: ValueError  [0.473s]
> sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_cpu_assignment  ->  passed  [0.348s]
> sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_deferred_requests  ->  failed: /usr/tests/atf_python/sys/netlink/netlink.py:376: ValueError  [0.480s]
> sys/netinet/tcp_hpts_test.py:TestTcpHpts::test_direct_wake_mechanism  ->  Sleeping on "-" with the following non-sleepable locks held:
> exclusive sleep mutex tcp_hpts_lck (hpts) r = 0 (0xfffff801a0f39b00) locked @ /usr/src/sys/netinet/tcp_hpts_test.c:1467
> stack backtrace:
> #0 0xffffffff80c061dc at witness_debugger+0x6c
> #1 0xffffffff80c073f0 at witness_warn+0x430
> #2 0xffffffff80b9a928 at _sleep+0x58
> #3 0xffffffff80bf8e91 at taskqueue_thread_loop+0xc1
> #4 0xffffffff80b3e472 at fork_exit+0x82
> #5 0xffffffff810b457e at fork_trampoline+0xe
> Sleeping on "-" with the following non-sleepable locks held:
> exclusive sleep mutex tcp_hpts_lck (hpts) r = 0 (0xfffff801a0f39b00) locked @ /usr/src/sys/netinet/tcp_hpts_test.c:1467
> stack backtrace:
> #0 0xffffffff80c061dc at witness_debugger+0x6c
> #1 0xffffffff80c073f0 at witness_warn+0x430
> #2 0xffffffff80b9a928 at _sleep+0x58
> #3 0xffffffff80bf8e91 at taskqueue_thread_loop+0xc1
> #4 0xffffffff80b3e472 at fork_exit+0x82
> #5 0xffffffff810b457e at fork_trampoline+0xe
> Sleeping thread (tid 104254, pid 0) owns a non-sleepable lock
> KDB: stack backtrace of thread 104254:
> sched_switch() at sched_switch+0x5e2/frame 0xfffffe00a2345dc0
> mi_switch() at mi_switch+0x172/frame 0xfffffe00a2345de0
> sleepq_switch() at sleepq_switch+0x109/frame 0xfffffe00a2345e20
> _sleep() at _sleep+0x2a5/frame 0xfffffe00a2345ec0
> taskqueue_thread_loop() at taskqueue_thread_loop+0xc1/frame 0xfffffe00a2345ef0
> fork_exit() at fork_exit+0x82/frame 0xfffffe00a2345f30
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a2345f30
> --- trap 0x37223630, rip = 0x1, rsp = 0, rbp = 0x1f093737d360 ---
> panic: sleeping thread holds tcp_hpts_lck
> cpuid = 0
> time = 1760426198
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a2304b20
> vpanic() at vpanic+0x136/frame 0xfffffe00a2304c50
> panic() at panic+0x43/frame 0xfffffe00a2304cb0
> propagate_priority() at propagate_priority+0x2a6/frame 0xfffffe00a2304cf0
> turnstile_wait() at turnstile_wait+0x399/frame 0xfffffe00a2304d30
> __mtx_lock_sleep() at __mtx_lock_sleep+0x1c1/frame 0xfffffe00a2304dc0
> __mtx_lock_flags() at __mtx_lock_flags+0xdd/frame 0xfffffe00a2304e10
> tcp_hpts_thread() at tcp_hpts_thread+0x2a/frame 0xfffffe00a2304e60
> ithread_loop() at ithread_loop+0x266/frame 0xfffffe00a2304ef0
> fork_exit() at fork_exit+0x82/frame 0xfffffe00a2304f30
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a2304f30
> --- trap 0xc, rip = 0x2fc0a3a8a9da, rsp = 0x2fc0a66f2f48, rbp = 0x2fc0a66f2f60 ---
> KDB: enter: panic
> [ thread pid 12 tid 104256 ]
> Stopped at      kdb_enter+0x33: movq    $0,0x121d0a2(%rip)
> db:0:kdb.enter.panic> show pcpu
> cpuid        = 0
> dynamic pcpu = 0xa075c0
> curthread    = 0xfffff8014de3c000: pid 12 tid 104256 critnest 1 "swi1: hpts"
> curpcb       = 0xfffff8014de3c550
> fpcurthread  = none
> idlethread   = 0xfffff8000460b780: tid 100003 "idle: cpu0"
> self         = 0xffffffff82610000
> curpmap      = 0xffffffff81d9f520
> tssp         = 0xffffffff82610384
> rsp0         = 0xfffffe00a2305000
> kcr3         = 0x800000000259b002
> ucr3         = 0xffffffffffffffff
> scr3         = 0x19ecf6834
> gs32p        = 0xffffffff82610404
> ldt          = 0xffffffff82610444
> tss          = 0xffffffff82610434
> curvnet      = 0
> spin locks held:
> db:0:kdb.enter.panic>  reset
> Uptime: 16m1s
> + rc=0
> + echo 'bhyve return code = 0'
> bhyve return code = 0
> + sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-26985' --destroy
> + sh -ex freebsd-ci/scripts/test/extract-meta.sh
> + METAOUTDIR=meta-out
> + rm -fr meta-out
> + mkdir meta-out
> + tar xvf meta.tar -C meta-out
> x ./
> x ./run-kyua.sh
> x ./auto-shutdown
> x ./disable-dtrace-tests.sh
> x ./run.sh
> x ./disable-zfs-tests.sh
> x ./disable-notyet-tests.sh
> + rm -f 'test-report.*'
> + mv 'meta-out/test-report.*' .
> mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
> + report=test-report.xml
> + [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
> + rm -f disk-cam
> + jot 5
> + rm -f disk1
> + rm -f disk2
> + rm -f disk3
> + rm -f disk4
> + rm -f disk5
> + rm -f disk-test.img
> [PostBuildScript] - [INFO] Executing post build scripts.
> [FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins9221278409060209048.sh
> + ./freebsd-ci/artifact/post-link.py
> Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '4fc1e7546f1289183ec5d0b80653cab090c60399', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
> "Link created: main/latest_tested/amd64/amd64 -> ../../4fc1e7546f1289183ec5d0b80653cab090c60399/amd64/amd64\n"
> Recording test results
> ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
> Checking for post-build
> Performing post-build step
> Checking if email needs to be generated
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> Sending mail from default account using System Admin e-mail address