FreeBSD-main-amd64-test - Build #24970 - Failure

Reply: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #24971 - Still unstable"
In reply to: jenkins-admin_a_FreeBSD.org: "FreeBSD-main-amd64-test - Build #24969 - Still Unstable"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: <jenkins-admin_at_FreeBSD.org>
Date: Thu, 21 Mar 2024 17:12:28 UTC
FreeBSD-main-amd64-test - Build #24970 (6b3db5d7793f57fa0020a78d61aae7ec318557cc) - Failure

Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/24970/
Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/24970/changes
Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/24970/console

Status explanation:
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
                  this is a notification to note that these changes have
                  not been fully tested by the CI system

Change summaries:
(Those commits are likely but not certainly responsible)

25429e274371eba9a90ea27cbf73e88329916e0d by jhb:
ddp: Clear active DDP buffer members to NULL to pacify an assertion

3d0a736796a99fe70be9de97beec8f10970c6905 by jhb:
tcp: Add a new kernel-only TCP_USE_DDP socket option

eba13bbc37ab4f45a8a3502d59c37d56d9a04ca5 by jhb:
cxgbe: Support TCP_USE_DDP on offloaded TOE connections

a16ff32f04b5b891a2d9b0427a2fd9c48e866da3 by jhb:
NFS: Request use of TCP_USE_DDP for in-kernel TCP sockets

9978c6289d621ac9edc95acb4e0f527a62a49b03 by jhb:
cxgbe tom: Handle a race condition when enabling TLS offload

bcd401b5a39c6186bb1fe450286299f2a481244c by dim:
Merge commit bbb8a0df7367 from llvm-project (by Shafik Yaghmour):

55e2187a091b7c397b3682b5ded2143c43e31a18 by kib:
Translate linux_newlstat name argument

6b3db5d7793f57fa0020a78d61aae7ec318557cc by kevans:
libsysdecode: fix build after 3d0a736796a9



The end of the build log:

[...truncated 3.81 MiB...]
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (vhid removed)
epair0b: promiscuous mode disabled
epair1a: link state changed to DOWN
carp: 1@epair1b: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 240 (interface down)
epair1b: link state changed to DOWN
carp: demoted by -240 to 0 (vhid removed)
epair1b: promiscuous mode disabled
passed  [5.597s]
sys/netinet/carp:unicast_v6  ->  bridge0: Ethernet address: 58:9c:fc:10:05:21
epair0a: Ethernet address: 02:3c:b3:4f:44:0a
epair0b: Ethernet address: 02:3c:b3:4f:44:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair1a: Ethernet address: 02:90:67:24:ed:0a
epair1b: Ethernet address: 02:90:67:24:ed:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
bridge0: link state changed to UP
epair0a: promiscuous mode enabled
epair1a: promiscuous mode enabled
epair0b: promiscuous mode enabled
carp: 1@epair0b: INIT -> BACKUP (initialization complete)
epair1b: promiscuous mode enabled
carp: 1@epair1b: INIT -> BACKUP (initialization complete)
carp: 1@epair0b: BACKUP -> MASTER (master timed out)
epair1a: promiscuous mode disabled
epair0a: promiscuous mode disabled
bridge0: link state changed to DOWN
epair0a: link state changed to DOWN
carp: 1@epair0b: MASTER -> INIT (hardware interface down)
carp: demoted by 240 to 240 (interface down)
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (vhid removed)
epair0b: promiscuous mode disabled
epair1a: link state changed to DOWN
carp: 1@epair1b: BACKUP -> INIT (hardware interface down)
carp: demoted by 240 to 240 (interface down)
epair1b: link state changed to DOWN
carp: demoted by -240 to 0 (vhid removed)
epair1b: promiscuous mode disabled
passed  [5.534s]
sys/netinet/carp.py:TestCarp::test_source_mac  ->  epair0a: Ethernet address: 02:9f:2c:2f:8b:0a
epair0b: Ethernet address: 02:9f:2c:2f:8b:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
lo0: link state changed to UP
epair0a: promiscuous mode enabled
carp: 1@epair0a: INIT -> BACKUP (initialization complete)
carp: 1@epair0a: BACKUP -> MASTER (master timed out)
carp: 1@epair0a: MASTER -> INIT (hardware interface down)
carp: demoted by 240 to 240 (interface down)
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
carp: demoted by -240 to 0 (vhid removed)
epair0a: promiscuous mode disabled
lock order reversal:
 1st 0xffffffff81aa9b98 ifnet_sx (ifnet_sx, sx) @ /usr/src/sys/netinet/in.c:1227
 2nd 0xffffffff81aac930 in_control (in_control, sx) @ /usr/src/sys/netinet/in.c:359
lock order in_control -> ifnet_sx established at:
#0 0xffffffff80bc918a at witness_checkorder+0x2fa
#1 0xffffffff80b60381 at _sx_slock_int+0x61
#2 0xffffffff826d28db at sctp_init_ifns_for_vrf+0x2b
#3 0xffffffff826d2a59 at sctp_addr_change+0x49
#4 0xffffffff80cb5c41 at rt_addrmsg+0x121
#5 0xffffffff80d20a87 at in_addprefix+0x157
#6 0xffffffff80d20205 at in_control_ioctl+0xab5
#7 0xffffffff80dc8cff at rtnl_handle_addr+0x45f
#8 0xffffffff80dc6775 at rtnl_handle_message+0x195
#9 0xffffffff80dc4b17 at nl_taskqueue_handler+0x5e7
#10 0xffffffff80bbbd3b at taskqueue_run_locked+0xab
#11 0xffffffff80bbcdf3 at taskqueue_thread_loop+0xd3
#12 0xffffffff80b09aa2 at fork_exit+0x82
#13 0xffffffff8102e7be at fork_trampoline+0xe
lock order ifnet_sx -> in_control attempted at:
#0 0xffffffff80bc99f3 at witness_checkorder+0xb63
#1 0xffffffff80b5f22d at _sx_xlock+0x5d
#2 0xffffffff80d2129a at in_ifscrub_all+0xba
#3 0xffffffff80d3588b at ip_destroy+0x9b
#4 0xffffffff80cc9a74 at vnet_destroy+0x164
#5 0xffffffff80b134b0 at prison_deref+0x980
#6 0xffffffff80b145c7 at sys_jail_remove+0x117
#7 0xffffffff8105cb7c at amd64_syscall+0x85c
#8 0xffffffff8102e06b at fast_syscall_common+0xf8
passed  [4.376s]
sys/netinet/divert:ipdivert_ip_input_local_success  ->  epair0a: Ethernet address: 02:f6:cd:1d:78:0a
epair0b: Ethernet address: 02:f6:cd:1d:78:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0b: link state changed to DOWN
epair0a: link state changed to DOWN
passed  [1.113s]
sys/netinet/divert:ipdivert_ip_output_remote_success  ->  epair0a: Ethernet address: 02:0e:75:d2:dd:0a
epair0b: Ethernet address: 02:0e:75:d2:dd:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0b: link state changed to DOWN
epair0a: link state changed to DOWN
passed  [1.162s]
sys/netinet/fibs:fibs_ifroutes1_success  ->  epair0a: Ethernet address: 02:c5:3e:5a:b5:0a
epair0b: Ethernet address: 02:c5:3e:5a:b5:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
WARNING: Adding ifaddrs to all fibs has been turned off by default. Consider tuning net.add_addr_allfibs if needed
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
passed  [0.118s]
sys/netinet/fibs_test:arpresolve_checks_interface_fib  ->  skipped: firewalls interfere with this test  [0.042s]
sys/netinet/fibs_test:default_route_with_multiple_fibs_on_same_subnet  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
passed  [0.055s]
sys/netinet/fibs_test:default_route_with_multiple_fibs_on_same_subnet_inet6  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
passed  [0.055s]
sys/netinet/fibs_test:loopback_and_network_routes_on_nondefault_fib  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
passed  [0.055s]
sys/netinet/fibs_test:loopback_and_network_routes_on_nondefault_fib_inet6  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
passed  [0.055s]
sys/netinet/fibs_test:same_ip_multiple_ifaces  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
tap2: Ethernet address: 58:9c:fc:10:81:26
tap3: Ethernet address: 58:9c:fc:10:ff:d2
expected_failure: kern/189088 Assigning the same IP to multiple interfaces in different FIBs creates a host route for only one: atf-check failed; see the output of the test for details  [0.076s]
sys/netinet/fibs_test:same_ip_multiple_ifaces_fib0  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
tap2: Ethernet address: 58:9c:fc:10:81:26
tap3: Ethernet address: 58:9c:fc:10:ff:d2
passed  [0.056s]
sys/netinet/fibs_test:same_ip_multiple_ifaces_inet6  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
passed  [0.111s]
sys/netinet/fibs_test:slaac_on_nondefault_fib6  ->  skipped: firewalls interfere with this test  [0.053s]
sys/netinet/fibs_test:subnet_route_with_multiple_fibs_on_same_subnet  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
passed  [0.064s]
sys/netinet/fibs_test:subnet_route_with_multiple_fibs_on_same_subnet_inet6  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83
passed  [0.062s]
sys/netinet/fibs_test:udp_dontroute  ->  tap0: Ethernet address: 58:9c:fc:10:35:7f
tap1: Ethernet address: 58:9c:fc:10:ff:83


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff8270428e
stack pointer	        = 0x28:0xfffffe00a6eb38f0
frame pointer	        = 0x28:0xfffffe00a6eb3960
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (netlink_socket (PID)
rdi: fffff80003969d00 rsi: fffff8021cd80800 rdx: 000000000000004c
rcx: fffffe011e45f540  r8: fffff80003969fd8  r9: 0000000003969d00
rax: deadc0dedeadc0de rbx: 0000000000000000 rbp: fffffe00a6eb3960
r10: fffff8023fd83780 r11: 0000000000000004 r12: fffff80217fbcb10
r13: fffff80003969d00 r14: fffff80003969f00 r15: fffff80003932780
trap number		= 9
panic: general protection fault
cpuid = 0
time = 1711041147
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a6eb3630
vpanic() at vpanic+0x135/frame 0xfffffe00a6eb3760
panic() at panic+0x43/frame 0xfffffe00a6eb37c0
trap_fatal() at trap_fatal+0x40f/frame 0xfffffe00a6eb3820
calltrap() at calltrap+0x8/frame 0xfffffe00a6eb3820
--- trap 0x9, rip = 0xffffffff8270428e, rsp = 0xfffffe00a6eb38f0, rbp = 0xfffffe00a6eb3960 ---
sctp_add_addr_to_vrf() at sctp_add_addr_to_vrf+0x44e/frame 0xfffffe00a6eb3960
sctp_addr_change() at sctp_addr_change+0xde/frame 0xfffffe00a6eb39b0
rt_addrmsg() at rt_addrmsg+0x121/frame 0xfffffe00a6eb3a80
in_addprefix() at in_addprefix+0x157/frame 0xfffffe00a6eb3ad0
in_control_ioctl() at in_control_ioctl+0xab5/frame 0xfffffe00a6eb3b70
rtnl_handle_addr() at rtnl_handle_addr+0x45f/frame 0xfffffe00a6eb3cd0
rtnl_handle_message() at rtnl_handle_message+0x195/frame 0xfffffe00a6eb3d30
nl_taskqueue_handler() at nl_taskqueue_handler+0x5e7/frame 0xfffffe00a6eb3e40
taskqueue_run_locked() at taskqueue_run_locked+0xab/frame 0xfffffe00a6eb3ec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xd3/frame 0xfffffe00a6eb3ef0
fork_exit() at fork_exit+0x82/frame 0xfffffe00a6eb3f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00a6eb3f30
--- trap 0x656c706d, rip = 0x6c65637865, rsp = 0x200000018, rbp = 0x6f6b2e61303933 ---
KDB: enter: panic
[ thread pid 0 tid 102642 ]
Stopped at      kdb_enter+0x33: movq    $0,0xe38372(%rip)
db:0:kdb.enter.panic> show pcpu
cpuid        = 0
dynamic pcpu = 0x9f2040
curthread    = 0xfffff802174f1000: pid 0 tid 102642 critnest 1 "netlink_socket (PID"
curpcb       = 0xfffff802174f1520
fpcurthread  = none
idlethread   = 0xfffff8000387a740: tid 100003 "idle: cpu0"
self         = 0xffffffff82210000
curpmap      = 0xffffffff8197feb0
tssp         = 0xffffffff82210384
rsp0         = 0xfffffe00a6eb4000
kcr3         = 0x800000000216b003
ucr3         = 0xffffffffffffffff
scr3         = 0x21c1b2947
gs32p        = 0xffffffff82210404
ldt          = 0xffffffff82210444
tss          = 0xffffffff82210434
curvnet      = 0xfffff80003636000
spin locks held:
db:0:kdb.enter.panic>  reset
Uptime: 12m46s
+ rc=0
+ echo 'bhyve return code = 0'
bhyve return code = 0
+ sudo /usr/sbin/bhyvectl '--vm=testvm-main-amd64-24970' --destroy
+ sh -ex freebsd-ci/scripts/test/extract-meta.sh
+ METAOUTDIR=meta-out
+ rm -fr meta-out
+ mkdir meta-out
+ tar xvf meta.tar -C meta-out
x ./
x ./run.sh
x ./run-kyua.sh
x ./disable-dtrace-tests.sh
x ./auto-shutdown
x ./disable-zfs-tests.sh
x ./disable-notyet-tests.sh
+ rm -f test-report.txt test-report.xml
+ mv 'meta-out/test-report.*' .
mv: rename meta-out/test-report.* to ./test-report.*: No such file or directory
+ report=test-report.xml
+ [ -e freebsd-ci/jobs/FreeBSD-main-amd64-test/xfail-list -a -e test-report.xml ]
+ rm -f disk-cam
+ jot 5
+ rm -f disk1
+ rm -f disk2
+ rm -f disk3
+ rm -f disk4
+ rm -f disk5
+ rm -f disk-test.img
[PostBuildScript] - [INFO] Executing post build scripts.
[FreeBSD-main-amd64-test] $ /bin/sh -xe /tmp/jenkins2370642083270534177.sh
+ ./freebsd-ci/artifact/post-link.py
Post link: {'job_name': 'FreeBSD-main-amd64-test', 'commit': '6b3db5d7793f57fa0020a78d61aae7ec318557cc', 'branch': 'main', 'target': 'amd64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
"Link created: main/latest_tested/amd64/amd64 -> ../../6b3db5d7793f57fa0020a78d61aae7ec318557cc/amd64/amd64\n"
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Checking for post-build
Performing post-build step
Checking if email needs to be generated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending mail from default account using System Admin e-mail address