From nobody Mon Sep 12 12:10:44 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MR63m2qfGz4cRLT; Mon, 12 Sep 2022 12:53:56 +0000 (UTC) (envelope-from dsl@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MR63m2S95z3nkT; Mon, 12 Sep 2022 12:53:56 +0000 (UTC) (envelope-from dsl@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662987236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vLp+XGXa+qCOi+kDCBACiyV2vhs4+LMtmCGw8l06dyY=; b=nXh15A1NCc0YSrlHe3bayr7yMVOtXDXweM9csa1yWgbizB+ZsMOnxqo2cPlzKWIgLqy7w/ IBr+3SEwkp5Qy7XbAnOE3vD0QpGiiW8ZRK7U8jFGbuALB1jj++f90yLdB5TqrUfsEzee9q zXVO+HeCb5pse+gqzaP4qkfFYjDFJLGsoxKOURV5CJUgarzJxTPpM+WQtNhZc0wjDi0IaB cH7qVys4wdkTqFRGa1vKwj+0TLqPiSUkddalRkhlLoDRYrbiP04262ivsh/oAvkPXYbHbH KqsIoCaXjhVai2TLygq21lCQFF0i/lT9NbwDZgLyr6n7djFOtfJKI1WqSEXsTA== Received: from localhost (bmj149.neoplus.adsl.tpnet.pl [83.28.229.149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: dsl) by smtp.freebsd.org (Postfix) with ESMTPSA id 4MR63l3KrnzTrr; Mon, 12 Sep 2022 12:53:55 +0000 (UTC) (envelope-from dsl@FreeBSD.org) References: <1800459695.1.1646649539521@mailrelay> <132978150.92.1646660769467@mailrelay> <3374E0F8-D712-4ED0-A62B-B6924FC8A5E2@fubar.geek.nz> <20220308154204.GA37265@www.zefox.net> User-agent: mu4e 1.6.10; emacs 28.1 From: Dmitry Salychev To: bob prohaska Cc: Mark Johnston , Andrew Turner , Ronald Klop , Mark Millard , freebsd-arm@freebsd.org, freebsd-current@freebsd.org Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) Date: Mon, 12 Sep 2022 14:10:44 +0200 In-reply-to: <20220308154204.GA37265@www.zefox.net> Message-ID: <86czc0eotc.fsf@peasant.tower.home> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662987236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vLp+XGXa+qCOi+kDCBACiyV2vhs4+LMtmCGw8l06dyY=; b=qT3NCvCwgF6PbUkcfLsb3AKWznrCCvQo0vIl27bGcrnRy1vvHNnsHiZB0YJenX9SbvQMl5 gpBKz6V1SIT0frhyKRdeu6NV886MeA3n7Q5BzPD9NnPriI54sjQLaa9s/6CcEym3k/Clwd GyhjbS7d+BRDxetj/xidgZDZ3AcUo4Y8zOFasAW8rwjwkhNmWscUiYMWjxbfXidr0lMJ5Q j0yxEcLpFpE8lCuafWC34aYlG3e5qTeaCWq4O4Zki4k1792qbNwBfgt22jNwDputpvHeRB a3MYdkrYibDEqk8KZIk8wZ1IXI35T5DoFDxiLeEEJUxyNGJH6V6b+GY5u00m4A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1662987236; a=rsa-sha256; cv=none; b=fSnK2WajqNveM8tkHybFOtK0HeGsPpnRbt6WJyNvOB93uehPQe0I0vBmXYF9dEOjca2VAV pleLTGJ0465zvP92TgC3dWa9nwgL25t6xrSHHj7JM4HRxp3tjNSqSnT/stiVzUyojAzdw/ sW2AizgiWm1k2Z8O2tsyTP0QGUDmoXSdsaHdjuUHFSKwQWa0OKw6dO5Rf3pQ3lsi/KtPpm 4cAgYe9tk/LU+FRV/PluivGFG2DRuBKfgFBMSpwlX29JOeQFBFq5B+3ftlHC2lu7wcqJ0o NtYD2gvwJn9E9plzYyw1+GTx2CXJkKJVy24xowOEpZs1/kQqI8tNGFpvSzurfQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=dpaa2_panics.txt (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff000000460268 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff000000460018 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000006d8000, frame=0xffff0000b23a24c0) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 0xffff000000760d40 in arm_gic_v3_intr (arg=0xffffa000021ab000) at /usr/src/sys/arm64/arm64/gic_v3.c:505 #7 0xffff00000074be4c in intr_irq_handler (tf=0xffff0000b23a26e0) at /usr/src/sys/kern/subr_intr.c:327 #8 0xffff00000074be4c in intr_irq_handler (tf=0xffffa000006d8000) at /usr/src/sys/kern/subr_intr.c:327 #9 #10 cpu_idle (busy=) at /usr/src/sys/arm64/arm64/machdep.c:257 #11 0xffff000000491ab4 in sched_idletd (dummy=dummy@entry=0x0) at /usr/src/sys/kern/sched_ule.c:3070 #12 0xffff000000416440 in fork_exit (callout=0xffff00000049163c , arg=0x0, frame=0xffff0000b23a2990) at /usr/src/sys/kern/kern_fork.c:1102 #13 --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff000000460268 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff000000460018 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000024b5600, frame=0xffff0000bcda29c0) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 0xffff0000004ced5c in witness_lock (lock=lock@entry=0xffffa0000056ca20, flags=8, file=file@entry=0xffff00000093011e "/usr/src/sys/dev/dpaa2/dpaa2_swp.c", line=line@entry=795) at /usr/src/sys/kern/subr_witness.c:1505 #7 0xffff00000043a3a8 in __mtx_lock_flags (c=0xffffa0000056ca38, opts=0, file=0xffff00000093011e "/usr/src/sys/dev/dpaa2/dpaa2_swp.c", line=795) at /usr/src/sys/kern/kern_mutex.c:290 #8 0xffff0000007d60a8 in dpaa2_swp_enq_mult (swp=swp@entry=0xffffa0000056ca00, ed=ed@entry=0xffff0000bcda2c70, fd=fd@entry=0xffff0000bcda2df8, flags=flags@entry=0xffff0000bcda2c6c, frames_n=frames_n@entry=1) at /usr/src/sys/dev/dpaa2/dpaa2_swp.c:795 #9 0xffff0000007d445c in dpaa2_io_enq_multiple_fq (iodev=, fqid=, fd=0xffff00000093011e, frames_n=1473863424) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:343 #10 0xffff0000007d7e3c in DPAA2_SWP_ENQ_MULTIPLE_FQ (fqid=205, fd=0xffff0000bcda2df8, frames_n=1, dev=) at ./dpaa2_swp_if.h:45 #11 dpaa2_ni_tx (sc=, tx=, m=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2717 #12 dpaa2_ni_transmit (ifp=, m=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2374 #13 0xffff0000005878bc in ether_output_frame (ifp=ifp@entry=0xffffa00002550800, m=0x8, m@entry=0xffffa00032552900) at /usr/src/sys/net/if_ethersubr.c:513 #14 0xffff0000005876f4 in ether_output (ifp=0xffffa00002550800, m=0xffffa00032552900, dst=0xffffa00028010b68, ro=0xffffa00028010b48) at /usr/src/sys/net/if_ethersubr.c:436 #15 0xffff0000005dff2c in ip_output (m=0xffff00000093011e, m@entry=0xffffa00032552900, opt=, ro=0xffffa00028010b48, flags=, imo=0x0, inp=0xffffa000280109f0) at /usr/src/sys/netinet/ip_output.c:793 #16 0xffff0000005f7204 in tcp_default_output (tp=0xffff000111f72590) at /usr/src/sys/netinet/tcp_output.c:1542 #17 0xffff0000005f0ca4 in tcp_output (tp=0xffff000111f72590) at /usr/src/sys/netinet/tcp_var.h:407 #18 tcp_do_segment (m=0xffffa08004b26d00, th=0xffff00011d8ec0e2, so=0xffff000111f27d80, tp=0xffff000111f72590, drop_hdrlen=52, tlen=, iptos=) at /usr/src/sys/netinet/tcp_input.c:3294 #19 0xffff0000005ee094 in tcp_input_with_port (mp=, offp=, proto=, port=0) at /usr/src/sys/netinet/tcp_input.c:1425 #20 0xffff0000005eee70 in tcp_input (mp=0xffffa0000056ca20, offp=0x8, proto=9634078) at /usr/src/sys/netinet/tcp_input.c:1520 #21 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838 #22 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1153 #23 0xffff0000005a44c0 in netisr_dispatch (proto=5687840, m=0xffff00000093011e) at /usr/src/sys/net/netisr.c:1244 #24 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002550800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:925 #25 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002550800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:711 #26 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:741 #27 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1153 #28 0xffff0000005a44c0 in netisr_dispatch (proto=5687840, proto@entry=5, m=0xffff00000093011e, m@entry=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1244 #29 0xffff000000587f24 in ether_input (ifp=0xffffa00002550800, m=0xffffa08004b26d00) at /usr/src/sys/net/if_ethersubr.c:832 #30 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000be8f5000, fq=, fd=0xffff0000befb1360) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870 #31 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000be8f5000, src=, consumed=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763 #32 dpaa2_ni_poll (arg=0xffff0000be8f5000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633 #33 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c7c00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540 #34 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576200, p=) at /usr/src/sys/kern/kern_intr.c:1205 #35 ithread_execute_handlers (ie=0xffffa00000576200, p=) at /usr/src/sys/kern/kern_intr.c:1218 #36 ithread_loop (arg=, arg@entry=0xffffa000006c0a00) at /usr/src/sys/kern/kern_intr.c:1306 #37 0xffff000000416440 in fork_exit (callout=0xffff000000419cac , arg=0xffffa000006c0a00, frame=0xffff0000bcda3990) at /usr/src/sys/kern/kern_fork.c:1102 #38 --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff000000460268 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff000000460018 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000024b5600, frame=0xffff0000bcda2db0) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 0xffff0000005dee5c in ip_output (m=, m@entry=0xffffa00007036700, opt=, ro=0xffffa0006d1f5e98, flags=0, imo=0x0, inp=0xffffa0006d1f5d40) at /usr/src/sys/sys/mbuf.h:1533 #7 0xffff0000005f7204 in tcp_default_output (tp=0xffff00011e090450) at /usr/src/sys/netinet/tcp_output.c:1542 #8 0xffff0000005f0ca4 in tcp_output (tp=0xffff00011e090450) at /usr/src/sys/netinet/tcp_var.h:407 #9 tcp_do_segment (m=0xffffa00007036700, th=0xffff00011d71e0e2, so=0xffff00011cd8d680, tp=0xffff00011e090450, drop_hdrlen=52, tlen=, iptos=) at /usr/src/sys/netinet/tcp_input.c:3294 #10 0xffff0000005ee094 in tcp_input_with_port (mp=, offp=, proto=, port=0) at /usr/src/sys/netinet/tcp_input.c:1425 #11 0xffff0000005eee70 in tcp_input (mp=0xffffa00007036770, offp=0x1, proto=9323951) at /usr/src/sys/netinet/tcp_input.c:1520 #12 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838 #13 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1153 #14 0xffff0000005a44c0 in netisr_dispatch (proto=117663600, m=0xffff0000008e45af) at /usr/src/sys/net/netisr.c:1244 #15 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002550800, m=0x1) at /usr/src/sys/net/if_ethersubr.c:925 #16 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002550800, m=0x1) at /usr/src/sys/net/if_ethersubr.c:711 #17 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:741 #18 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1153 #19 0xffff0000005a44c0 in netisr_dispatch (proto=117663600, proto@entry=5, m=0xffff0000008e45af, m@entry=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1244 #20 0xffff000000587f24 in ether_input (ifp=0xffffa00002550800, m=0xffffa00007036700) at /usr/src/sys/net/if_ethersubr.c:832 #21 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000be8f5000, fq=, fd=0xffff0000befb1260) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870 #22 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000be8f5000, src=, consumed=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763 #23 dpaa2_ni_poll (arg=0xffff0000be8f5000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633 #24 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c7c00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540 #25 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576200, p=) at /usr/src/sys/kern/kern_intr.c:1205 #26 ithread_execute_handlers (ie=0xffffa00000576200, p=) at /usr/src/sys/kern/kern_intr.c:1218 #27 ithread_loop (arg=, arg@entry=0xffffa000006c0a00) at /usr/src/sys/kern/kern_intr.c:1306 #28 0xffff000000416440 in fork_exit (callout=0xffff000000419cac , arg=0xffffa000006c0a00, frame=0xffff0000bcda3990) at /usr/src/sys/kern/kern_fork.c:1102 #29 --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff000000460268 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff000000460018 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077c05c in do_el1h_sync (td=0xffffa08070a06600, frame=0xffff000111df33d0) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 0xffff0000004ced5c in witness_lock (lock=lock@entry=0xffff00011d2c2420, flags=8, file=file@entry=0xffff000000860324 "/usr/src/sys/kern/uipc_socket.c", line=line@entry=2240) at /usr/src/sys/kern/subr_witness.c:1505 #7 0xffff00000043a3a8 in __mtx_lock_flags (c=0xffff00011d2c2438, opts=0, file=0xffff000000860324 "/usr/src/sys/kern/uipc_socket.c", line=2240) at /usr/src/sys/kern/kern_mutex.c:290 #8 0xffff000000508f54 in soreceive_generic (so=0xffff00011d2c2200, psa=0x0, uio=, mp0=, controlp=0x0, flagsp=) at /usr/src/sys/kern/uipc_socket.c:2240 #9 0xffff00000050a57c in soreceive (so=0xffff00011d2c2420, psa=0x8, uio=0xffff000000860324, mp0=0x8c0, controlp=0xffffa08157d8c800, flagsp=0x10) at /usr/src/sys/kern/uipc_socket.c:2839 #10 0xffff0000004d37f4 in fo_read (fp=0xffffa0002f5215f0, uio=0xffff000111df3780, active_cred=0xffff000000860324, flags=0, td=0xffffa08070a06600) at /usr/src/sys/sys/file.h:341 #11 dofileread (td=td@entry=0xffffa08070a06600, fd=fd@entry=5, fp=0xffffa0002f5215f0, auio=auio@entry=0xffff000111df3780, offset=, flags=0) at /usr/src/sys/kern/sys_generic.c:369 #12 0xffff0000004d3460 in kern_readv (td=0xffffa08070a06600, fd=5, auio=auio@entry=0xffff000111df3780) at /usr/src/sys/kern/sys_generic.c:290 #13 0xffff0000004d3400 in sys_read (td=0xffff00011d2c2420, uap=) at /usr/src/sys/kern/sys_generic.c:206 #14 0xffff00000077c8c4 in syscallenter (td=0xffffa08070a06600) at /usr/src/sys/arm64/arm64/../../kern/subr_syscall.c:189 #15 svc_handler (td=0xffffa08070a06600, frame=) at /usr/src/sys/arm64/arm64/trap.c:199 #16 do_el0_sync (td=0xffffa08070a06600, frame=) at /usr/src/sys/arm64/arm64/trap.c:603 #17 #18 0x00000000859c9ef0 in ?? () #19 0x0000000088c00000 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff000000460268 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff000000460018 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000022b3000, frame=0xffff0000bcd79f90) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 0xffff0000004ced5c in witness_lock (lock=0xffff000000e31b00 , flags=8, file=0xffff0000008e30ac "/usr/src/sys/kern/kern_timeout.c", line=582) at /usr/src/sys/kern/subr_witness.c:1505 #7 0xffff00000047d4ec in callout_lock (c=0xffff0001121792c0) at /usr/src/sys/kern/kern_timeout.c:582 #8 callout_reset_sbt_on (c=0xffff0001121792c0, sbt=, prec=, ftn=0xffff00000047d4ec , arg=0xffff000112179000, cpu=0, flags=256) at /usr/src/sys/kern/kern_timeout.c:962 #9 0xffff0000005f0480 in tcp_do_segment (m=0xffffa08089e89c00, th=0xffff00011c8150e2, so=0xffff00011cd40b00, tp=, drop_hdrlen=52, tlen=, iptos=) at /usr/src/sys/netinet/tcp_input.c:2913 #10 0xffff0000005ee094 in tcp_input_with_port (mp=, offp=, proto=, port=0) at /usr/src/sys/netinet/tcp_input.c:1425 #11 0xffff0000005eee70 in tcp_input (mp=0xffff000000e31b00 , offp=0x8, proto=9318572) at /usr/src/sys/netinet/tcp_input.c:1520 #12 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838 #13 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1153 #14 0xffff0000005a44c0 in netisr_dispatch (proto=14883584, m=0xffff0000008e30ac) at /usr/src/sys/net/netisr.c:1244 #15 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002350800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:925 #16 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002350800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:711 #17 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:741 #18 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1153 #19 0xffff0000005a44c0 in netisr_dispatch (proto=14883584, proto@entry=5, m=0xffff0000008e30ac, m@entry=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1244 #20 0xffff000000587f24 in ether_input (ifp=0xffffa00002350800, m=0xffffa08089e89c00) at /usr/src/sys/net/if_ethersubr.c:832 #21 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000bf737000, fq=, fd=0xffff0000bfdf30a0) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870 #22 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000bf737000, src=, consumed=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763 #23 dpaa2_ni_poll (arg=0xffff0000bf737000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633 #24 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c6180) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540 #25 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576500, p=) at /usr/src/sys/kern/kern_intr.c:1205 #26 ithread_execute_handlers (ie=0xffffa00000576500, p=) at /usr/src/sys/kern/kern_intr.c:1218 #27 ithread_loop (arg=, arg@entry=0xffffa000006c0a60) at /usr/src/sys/kern/kern_intr.c:1306 #28 0xffff000000416440 in fork_exit (callout=0xffff000000419cac , arg=0xffffa000006c0a60, frame=0xffff0000bcd7a990) at /usr/src/sys/kern/kern_fork.c:1102 #29 --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff00000045d214 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff00000045cfc4 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001d61000, frame=0xffff0000bc7b26a0) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 sleepq_switch (wchan=0xffffa00000724780, wchan@entry=0x0, pri=0, pri@entry=7489536) at /usr/src/sys/kern/subr_sleepqueue.c:613 #7 0xffff0000004b8994 in sleepq_wait (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:660 #8 0xffff000000469134 in _sleep (ident=0xffffa00000724780, lock=0xffffa00000724800, priority=0, wmesg=0xffff000000885cee "-", sbt=0, pr=0, flags=256) at /usr/src/sys/kern/kern_synch.c:225 #9 0xffff0000004bed64 in TQ_SLEEP (tq=0xffffa00000724780, p=0xffffa00000724780, wm=) at /usr/src/sys/kern/subr_taskqueue.c:126 #10 taskqueue_thread_loop (arg=arg@entry=0xffff0000bd74e5a0) at /usr/src/sys/kern/subr_taskqueue.c:834 #11 0xffff000000413568 in fork_exit (callout=0xffff0000004beca8 , arg=0xffff0000bd74e5a0, frame=0xffff0000bc7b2990) at /usr/src/sys/kern/kern_fork.c:1102 #12 --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff00000045d214 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff00000045cfc4 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001ab8600, frame=0xffff0000bc7a2f50) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 mb_dtor_mbuf (mem=0xffffa0001d418800, size=256, arg=0x0) at /usr/src/sys/kern/kern_mbuf.c:681 #7 0xffff000000703e4c in item_dtor (zone=zone@entry=0xffff000041b5f000, item=item@entry=0xffffa0001d418800, size=256, udata=udata@entry=0x0, skip=) at /usr/src/sys/vm/uma_core.c:3488 #8 0xffff000000702a08 in uma_zfree_arg (zone=0xffff000041b5f000, item=item@entry=0xffffa0001d418800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4493 #9 0xffff0000004332cc in mb_free_ext (m=m@entry=0xffffa0001d418800) at /usr/src/sys/kern/kern_mbuf.c:1212 #10 0xffff000000431c2c in m_free (m=0xffffa0001d418800) at /usr/src/sys/sys/mbuf.h:1523 #11 0xffff00000043303c in m_freem (mb=0xffff000000432194 , mb@entry=0xffff00011d95f590) at /usr/src/sys/kern/kern_mbuf.c:1571 #12 0xffff0000005012fc in sbdrop (sb=sb@entry=0xffff00011c921310, len=, len@entry=2896) at /usr/src/sys/kern/uipc_sockbuf.c:1654 #13 0xffff0000005ebfec in tcp_do_segment (m=0xffffa08148514300, th=0xffff00011bbee0e2, so=0xffff00011c921000, tp=0xffff00011d95f590, drop_hdrlen=52, tlen=, iptos=) at /usr/src/sys/netinet/tcp_input.c:1850 #14 0xffff0000005e829c in tcp_input_with_port (mp=, offp=, proto=, port=0) at /usr/src/sys/netinet/tcp_input.c:1425 #15 0xffff0000005e9078 in tcp_input (mp=0xffffa0001d418800, offp=0x100, proto=0) at /usr/src/sys/netinet/tcp_input.c:1520 #16 0xffff0000005d60dc in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838 #17 0xffff00000059e3b0 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1153 #18 0xffff00000059e760 in netisr_dispatch (proto=490833920, m=0x0) at /usr/src/sys/net/netisr.c:1244 #19 0xffff000000581dcc in ether_demux (ifp=ifp@entry=0xffffa00001b57800, m=0x100) at /usr/src/sys/net/if_ethersubr.c:925 #20 0xffff000000583430 in ether_input_internal (ifp=0xffffa00001b57800, m=0x100) at /usr/src/sys/net/if_ethersubr.c:711 #21 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:741 #22 0xffff00000059e3b0 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1153 #23 0xffff00000059e760 in netisr_dispatch (proto=490833920, proto@entry=5, m=0x0, m@entry=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1244 #24 0xffff00000058228c in ether_input (ifp=0xffffa00001b57800, m=0xffffa08148514300) at /usr/src/sys/net/if_ethersubr.c:832 #25 0xffff0000007d3bc4 in dpaa2_ni_rx (chan=0xffff0000be1d1000, fq=, fd=0xffff0000be88d060) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2879 #26 0xffff0000007d37a8 in dpaa2_ni_consume_frames (chan=0xffff0000be1d1000, src=, consumed=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2770 #27 dpaa2_ni_poll (arg=0xffff0000be1d1000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633 #28 0xffff0000007cc2a8 in dpaa2_io_intr (arg=0xffffa000006bfc00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540 #29 0xffff00000041707c in intr_event_execute_handlers (ie=0xffffa0000056e200, p=) at /usr/src/sys/kern/kern_intr.c:1205 #30 ithread_execute_handlers (ie=0xffffa0000056e200, p=) at /usr/src/sys/kern/kern_intr.c:1218 #31 ithread_loop (arg=, arg@entry=0xffffa000006b8a00) at /usr/src/sys/kern/kern_intr.c:1306 #32 0xffff000000413568 in fork_exit (callout=0xffff000000416dd4 , arg=0xffffa000006b8a00, frame=0xffff0000bc7a3990) at /usr/src/sys/kern/kern_fork.c:1102 #33 --- Undefined instruction: f9401bf9 x0: 0 x1: ffffa0000073ae50 x2: 1 x3: 0 x4: 0 x5: 1 x6: 0 x7: ffff0000bc795c8c (_end + bb7e4c8c) x8: 9de11000 x9: 0 x10: 0 x11: 1ff x12: 5a8 x13: 42 x14: 9ac68042 x15: 1 x16: 10000 x17: 83cdd00 x18: ffff0000bc795c50 (_end + bb7e4c50) x19: 1d x20: ffff0000bdebfc18 (_end + bcf0ec18) x21: ffffa00043275a00 x22: ffff0000bd478000 (_end + bc4c7000) x23: ffff0000bdebfc20 (_end + bcf0ec20) x24: ffff0000bdebf1a0 (_end + bcf0e1a0) x25: ffff0000c0211000 (_end + bf260000) x26: ffff0000bde0e000 (_end + bce5d000) x27: ffff0000bde0e000 (_end + bce5d000) x28: ffff0000bdebfc08 (_end + bcf0ec08) x29: ffff0000bc795e20 (_end + bb7e4e20) sp: ffff0000bc795c50 lr: ffff0000007cf584 (dpaa2_ni_transmit + 2bc) elr: ffff00000049c5c0 (bus_dmamap_load + 114) spsr: 80000045 far: 129a54403538 panic: Unknown kernel exception 0 esr_el1 2000000 cpuid = 3 time = 1662915749 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 do_el1h_sync() at do_el1h_sync+0x194 handle_el1h_sync() at handle_el1h_sync+0x10 --- exception, esr 0x2000000 bus_dmamap_load() at bus_dmamap_load+0x114 ether_output_frame() at ether_output_frame+0xd4 ether_output() at ether_output+0x640 ip_output() at ip_output+0x1264 tcp_default_output() at tcp_default_output+0x1eb4 tcp_do_segment() at tcp_do_segment+0x1da4 tcp_input_with_port() at tcp_input_with_port+0xc00 tcp_input() at tcp_input+0xc ip_input() at ip_input+0x30c netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_demux() at ether_demux+0x174 ether_nh_input() at ether_nh_input+0x3ec netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_input() at ether_input+0x80 dpaa2_ni_rx() at dpaa2_ni_rx+0x174 dpaa2_ni_poll() at dpaa2_ni_poll+0xb8 dpaa2_io_intr() at dpaa2_io_intr+0x13c ithread_loop() at ithread_loop+0x2a4 fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 12 tid 100104 ] Stopped at kdb_enter+0x44: undefined f900027f --- Undefined instruction: a8c17bfd x0: ffffa00001ab6000 x1: ffff0000bc78aaa0 (_end + bb7d9aa0) x2: ffff0000418f4300 (_end + 40943300) x3: 8e4 x4: ffff0000bc78aaa0 (_end + bb7d9aa0) x5: 0 x6: 0 x7: ffff0000bc78a5dc (_end + bb7d95dc) x8: 0 x9: 1 x10: 2710 x11: 7ff8d402 x12: 7ff8a9fb x13: 2af8 x14: 2a07 x15: 0 x16: 1 x17: 100 x18: ffff0000bc78a790 (_end + bb7d9790) x19: ffffa00001ab6000 x20: ffffa000006d0000 x21: ffff0000418f4300 (_end + 40943300) x22: ffff000000bb3000 (sdt_vfs_vop_vop_spare3_return + 58) x23: ffff000000e5ea58 (group + 40) x24: 0 x25: ffff0000bc78a7b8 (_end + bb7d97b8) x26: ffff000000e00000 (thread0_st + 380) x27: ffff000000e55000 (kdb_why + 0) x28: ffff000000c3ee80 (pcpu_entry_tdq + 0) x29: ffff0000bc78a790 (_end + bb7d9790) sp: ffff0000bc78a790 lr: ffff0000007734b4 (cpu_switch + 60) elr: ffff000000775540 (vfp_save_state + e0) spsr: c5 far: 8649a000 panic: Unknown kernel exception 0 esr_el1 2000000 cpuid = 7 time = 1662916027 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 do_el1h_sync() at do_el1h_sync+0x194 handle_el1h_sync() at handle_el1h_sync+0x10 --- exception, esr 0x2000000 vfp_save_state() at vfp_save_state+0xe0 cpu_switch() at cpu_switch+0x5c mi_switch() at mi_switch+0x184 ithread_loop() at ithread_loop+0x9c fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 12 tid 100100 ] Stopped at kdb_enter+0x44: undefined f900027f --- panic: mutex not owned at /usr/src/sys/kern/uipc_socket.c:2187 cpuid = 7 time = 1662916786 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 __mtx_assert() at __mtx_assert+0xf0 soreceive_generic() at soreceive_generic+0x54c soreceive() at soreceive+0x44 dofileread() at dofileread+0x7c kern_readv() at kern_readv+0x50 sys_read() at sys_read+0x80 do_el0_sync() at do_el0_sync+0x54c handle_el0_sync() at handle_el0_sync+0x40 --- exception, esr 0x56000000 KDB: enter: panic [ thread pid 1991 tid 100180 ] Stopped at kdb_enter+0x44: undefined f900027f --- Undefined instruction: b9805668 x0: 0 x1: ffffa00001ab8600 x2: ffff0000008998ad (do_execve.fexecv_proc_title + 10764) x3: 7e x4: 6d x5: cb x6: ffff0000bc7a2cd8 (_end + bb7f1cd8) x7: ffff0000bc7a2cd4 (_end + bb7f1cd4) x8: ffffa00001ab8600 x9: 2 x10: 0 x11: ffff000000ead678 (w_locklistdata + 3ff30) x12: 3 x13: 2 x14: 10000 x15: 1 x16: 10000 x17: f5fa4dfb x18: ffff0000bc7a2bc0 (_end + bb7f1bc0) x19: ffffa0001e44a400 x20: ffff0000bc7a2cd4 (_end + bb7f1cd4) x21: ffff0000bc7a2cd8 (_end + bb7f1cd8) x22: ffffa0003b0379a4 x23: 42 x24: ffffa00001b5f400 x25: ffffa0003b037962 x26: 6c0 x27: ffffa0003b038022 x28: 962 x29: ffff0000bc7a2be0 (_end + bb7f1be0) sp: ffff0000bc7a2bc0 lr: ffff00000074da20 (bounce_bus_dmamap_load_buffer + d4) elr: ffff00000074db1c (bounce_bus_dmamap_load_buffer + 1d0) spsr: 20000045 far: 82f3fdac panic: Unknown kernel exception 0 esr_el1 2000000 cpuid = 4 time = 1662917050 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 do_el1h_sync() at do_el1h_sync+0x194 handle_el1h_sync() at handle_el1h_sync+0x10 --- exception, esr 0x2000000 bounce_bus_dmamap_load_buffer() at bounce_bus_dmamap_load_buffer+0x1d0 bus_dmamap_load_mbuf_sg() at bus_dmamap_load_mbuf_sg+0x88 dpaa2_ni_transmit() at dpaa2_ni_transmit+0x190 ether_output_frame() at ether_output_frame+0xd4 ether_output() at ether_output+0x640 ip_output() at ip_output+0x1264 tcp_default_output() at tcp_default_output+0x1eb4 tcp_output() at tcp_output+0x38 tcp_do_segment() at tcp_do_segment+0x28d8 tcp_input_with_port() at tcp_input_with_port+0xc00 tcp_input() at tcp_input+0xc ip_input() at ip_input+0x30c netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_demux() at ether_demux+0x174 ether_nh_input() at ether_nh_input+0x3ec netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_input() at ether_input+0x80 dpaa2_ni_rx() at dpaa2_ni_rx+0x174 dpaa2_ni_poll() at dpaa2_ni_poll+0xb8 dpaa2_io_intr() at dpaa2_io_intr+0x13c ithread_loop() at ithread_loop+0x2a4 fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 12 tid 100103 ] Stopped at kdb_enter+0x44: undefined f900027f --- Fatal data abort: x0: ffffa00001ab6000 x1: ffff0000bc785fb8 (_end + bb7d4fb8) x2: ffff000000928fa8 (console_pausestr + 24fd1) x3: 7e x4: 7e x5: 60 x6: d28938770a080101 x7: fc7aaec8d2893877 x8: 0 x9: 2 x10: 2 x11: ffff000000eada30 (w_locklistdata + 402e8) x12: 3 x13: 2 x14: 0 x15: 1 x16: 0 x17: 28 x18: ffff0000bc786030 (_end + bb7d5030) x19: ffff00011c52ecf0 x20: ffff00011bb4a680 x21: ffff00011c52ed10 x22: ffffa0812fc9cd00 x23: ffff00011c52ed08 x24: 5a8 x25: ffffa0812fc9cd84 x26: ffff000000857a08 (cam_status_table + 4b60) x27: ffff000000bb3000 (sdt_vfs_vop_vop_spare3_return + 58) x28: 1 x29: ffff0000bc786190 (_end + bb7d5190) sp: ffff0000bc786030 lr: ffff0000005f126c (tcp_default_output + 1d18) elr: fffefffffc7f2b70 spsr: 20000045 far: fffefffffc7f2b70 esr: 86000004 panic: vm_fault failed: fffefffffc7f2b70 error 1 cpuid = 7 time = 1662983268 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x13c panic() at panic+0x44 data_abort() at data_abort+0x2ec handle_el1h_sync() at handle_el1h_sync+0x10 --- exception, esr 0x86000004 (null)() at 0xfffffffffc7f2b70 tcp_do_segment() at tcp_do_segment+0x1da4 tcp_input_with_port() at tcp_input_with_port+0xc00 tcp_input() at tcp_input+0xc ip_input() at ip_input+0x30c netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_demux() at ether_demux+0x174 ether_nh_input() at ether_nh_input+0x3ec netisr_dispatch_src() at netisr_dispatch_src+0xe0 ether_input() at ether_input+0x80 dpaa2_ni_rx() at dpaa2_ni_rx+0x174 dpaa2_ni_poll() at dpaa2_ni_poll+0x140 dpaa2_io_intr() at dpaa2_io_intr+0x13c ithread_loop() at ithread_loop+0x2a4 fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 12 tid 100100 ] Stopped at kdb_enter+0x44: undefined f900027f db> --- (kgdb) bt #0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36 #1 kdb_enter (why=, msg=) at /usr/src/sys/kern/subr_kdb.c:508 #2 0xffff00000045d214 in vpanic (fmt=, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967 #3 0xffff00000045cfc4 in panic (fmt=0x12 ) at /usr/src/sys/kern/kern_shutdown.c:903 #4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001ab6000, frame=0xffff0000bc785a30) at /usr/src/sys/arm64/arm64/trap.c:532 #5 #6 bounce_bus_dmamap_load_buffer (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, buf=0xffffa0005200ea62, buflen=66, pmap=0xffff000000fa1050 , flags=, segs=0xffff0000bc785cd8, segp=0xffff0000bc785cd4) at /usr/src/sys/arm64/arm64/busdma_bounce.c:868 #7 0xffff00000049c7c4 in _bus_dmamap_load_buffer (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, buf=0xffff0000008998ad, buflen=126, pmap=0xffff000000fa1050 , flags=2049, segs=0xffff0000bc785cd8, segp=0xffff0000bc785cd4) at /usr/src/sys/arm64/include/bus_dma.h:129 #8 _bus_dmamap_load_mbuf_sg (m0=, nsegs=0xffff0000bc785cd4, flags=1, dmat=, map=, segs=) at /usr/src/sys/kern/subr_bus_dma.c:252 #9 bus_dmamap_load_mbuf_sg (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, m0=, m0@entry=0xffffa0005200ea00, segs=segs@entry=0xffff0000bc785cd8, nsegs=nsegs@entry=0xffff0000bc785cd4, flags=flags@entry=1) at /usr/src/sys/kern/subr_bus_dma.c:530 #10 0xffff0000007cf45c in dpaa2_ni_tx (sc=0xffff0000bd478000, tx=0xffff0000bf0c51a0, m=0xffffa0005200ea00) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2683 #11 dpaa2_ni_transmit (ifp=, m=0xffffa0005200ea00) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2374 #12 0xffff000000581c24 in ether_output_frame (ifp=ifp@entry=0xffffa00001b57800, m=0xffffa00001ab6000, m@entry=0xffffa0005200ea00) at /usr/src/sys/net/if_ethersubr.c:513 #13 0xffff000000581a5c in ether_output (ifp=0xffffa00001b57800, m=0xffffa0005200ea00, dst=0xffffa000495734c8, ro=0xffffa000495734a8) at /usr/src/sys/net/if_ethersubr.c:436 #14 0xffff0000005da134 in ip_output (m=0xffff0000008998ad, m@entry=0xffffa0005200ea00, opt=, ro=0xffffa000495734a8, flags=, imo=0x0, inp=0xffffa00049573350) at /usr/src/sys/netinet/ip_output.c:793 #15 0xffff0000005f140c in tcp_default_output (tp=0xffff0000fe9fc000) at /usr/src/sys/netinet/tcp_output.c:1542 #16 0xffff0000005ec720 in tcp_output (tp=tp@entry=0xffff0000fe9fc000) at /usr/src/sys/netinet/tcp_var.h:407 #17 0xffff0000005eb9e0 in tcp_do_segment (m=, th=, so=0xffff00011c7d9680, tp=0xffff0000fe9fc000, drop_hdrlen=52, tlen=, iptos=) at /usr/src/sys/netinet/tcp_input.c:1963 #18 0xffff0000005e829c in tcp_input_with_port (mp=, offp=, proto=, port=0) at /usr/src/sys/netinet/tcp_input.c:1425 #19 0xffff0000005e9078 in tcp_input (mp=0x0, offp=0xffffa00001ab6000, proto=9017517) at /usr/src/sys/netinet/tcp_input.c:1520 #20 0xffff0000005d60dc in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838 #21 0xffff00000059e3b0 in netisr_dispatch_src (proto=1, source=0, m=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1153 #22 0xffff00000059e760 in netisr_dispatch (proto=0, m=0xffff0000008998ad) at /usr/src/sys/net/netisr.c:1244 #23 0xffff000000581dcc in ether_demux (ifp=ifp@entry=0xffffa00001b57800, m=0xffffa00001ab6000) at /usr/src/sys/net/if_ethersubr.c:925 #24 0xffff000000583430 in ether_input_internal (ifp=0xffffa00001b57800, m=0xffffa00001ab6000) at /usr/src/sys/net/if_ethersubr.c:711 #25 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:741 #26 0xffff00000059e3b0 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1153 #27 0xffff00000059e760 in netisr_dispatch (proto=0, proto@entry=5, m=0xffff0000008998ad, m@entry=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1244 #28 0xffff00000058228c in ether_input (ifp=0xffffa00001b57800, m=0xffffa00052009c00) at /usr/src/sys/net/if_ethersubr.c:832 #29 0xffff0000007d3be4 in dpaa2_ni_rx (chan=0xffff0000bf014000, fq=, fd=0xffff0000bf6d00a0) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2879 #30 0xffff0000007d37c8 in dpaa2_ni_consume_frames (chan=0xffff0000bf014000, src=, consumed=) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2770 #31 dpaa2_ni_poll (arg=0xffff0000bf014000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633 #32 0xffff0000007cc2c8 in dpaa2_io_intr (arg=0xffffa000006be180) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540 --Type for more, q to quit, c to continue without paging-- #33 0xffff00000041707c in intr_event_execute_handlers (ie=0xffffa0000056e500, p=) at /usr/src/sys/kern/kern_intr.c:1205 #34 ithread_execute_handlers (ie=0xffffa0000056e500, p=) at /usr/src/sys/kern/kern_intr.c:1218 #35 ithread_loop (arg=, arg@entry=0xffffa000006b8a60) at /usr/src/sys/kern/kern_intr.c:1306 #36 0xffff000000413568 in fork_exit (callout=0xffff000000416dd4 , arg=0xffffa000006b8a60, frame=0xffff0000bc786990) at /usr/src/sys/kern/kern_fork.c:1102 #37 (kgdb) --=-=-= Content-Type: text/plain Hi, It seems that the recent 14-CURRENT/aarch64 (866e021) with DPAA2 drivers panics under network throughtput stress test in random places with unknown kernel exception 0 esr_el1 2000000 on Ten64 board (based on NXP's LS1088A, Cortex-A53), but the same code doesn't panic on HoneyComb (NXP LX2160A, Cortex-A72) even after ~10h long tests. I've gathered some stack backtraces from ddb and kgdb (attached). Panic itself can easily be reproduced after several minutes from the start of the test. I've tried to change PCPU_PTR macro to use get_pcpu again (as discussed in the thread earlier), but it didn't help. If you want to get your hands dirty, DPAA2 stuff I'm using is at https://github.com/mcusim/freebsd-src/tree/lx2160acex7-exp (branch is lx2160acex7-exp!) Any ideas or places to check would be really helpful. bob prohaska writes: > On Mon, Mar 07, 2022 at 11:45:02AM -0500, Mark Johnston wrote: >> On Mon, Mar 07, 2022 at 04:25:22PM +0000, Andrew Turner wrote: >> > >> > > On 7 Mar 2022, at 15:13, Mark Johnston wrote: >> > > ... >> > > A (the?) problem is that the compiler is treating "pc" as an alias >> > > for x18, but the rmlock code assumes that the pcpu pointer is loaded >> > > once, as it dereferences "pc" outside of the critical section. On >> > > arm64, if a context switch occurs between the store at _rm_rlock+144 and >> > > the load at +152, and the thread is migrated to another CPU, then we'll >> > > end up using the wrong CPU ID in the rm->rm_writecpus test. >> > > >> > > I suspect the problem is unique to arm64 as its get_pcpu() >> > > implementation is different from the others in that it doesn't use >> > > volatile-qualified inline assembly. This has been the case since >> > > https://cgit.freebsd.org/src/commit/?id=63c858a04d56529eddbddf85ad04fc8e99e73762 >> > > >> > > . >> > > >> > > I haven't been able to reproduce any crashes running poudriere in an >> > > arm64 AWS instance, though. Could you please try the patch below and >> > > confirm whether it fixes your panics? I verified that the apparent >> > > problem described above is gone with the patch. >> > >> > Alternatively (or additionally) we could do something like the following. There are only a few MI users of get_pcpu with the main place being in rm locks. >> > >> > diff --git a/sys/arm64/include/pcpu.h b/sys/arm64/include/pcpu.h >> > index 09f6361c651c..59b890e5c2ea 100644 >> > --- a/sys/arm64/include/pcpu.h >> > +++ b/sys/arm64/include/pcpu.h >> > @@ -58,7 +58,14 @@ struct pcpu; >> > >> > register struct pcpu *pcpup __asm ("x18"); >> > >> > -#define get_pcpu() pcpup >> > +static inline struct pcpu * >> > +get_pcpu(void) >> > +{ >> > + struct pcpu *pcpu; >> > + >> > + __asm __volatile("mov %0, x18" : "=&r"(pcpu)); >> > + return (pcpu); >> > +} >> > >> > static inline struct thread * >> > get_curthread(void) >> >> Indeed, I think this is probably the best solution. > > Just for fun I tried the patch on a Pi3 running -current, updated a day or two > prior. The patch applied, compiled and seemed to run acceptably, but when I > left a -j2 -DWITH_META_MODE buildworld running it crashed overnight, reporting > > > login: panic: rm_rlock: recursed on non-recursive rmlock sysctl lock @ /usr/src/sys/kern/kern_sysctl.c:193 > > cpuid = 0 > time = 1646720264 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x174 > panic() at panic+0x44 > _rm_rlock_debug() at _rm_rlock_debug+0x214 > sysctl_root_handler_locked() at sysctl_root_handler_locked+0x140 > sysctl_root() at sysctl_root+0x1ac > userland_sysctl() at userland_sysctl+0x140 > sys___sysctl() at sys___sysctl+0x68 > do_el0_sync() at do_el0_sync+0x520 > handle_el0_sync() at handle_el0_sync+0x40 > --- exception, esr 0x56000000 > KDB: enter: panic > [ thread pid 869 tid 100091 ] > Stopped at kdb_enter+0x44: undefined f902011f > > > I tried typing bt at the debugger prompt but got no more output. > > I've put the buildworld log file at > http://www.zefox.net/~fbsd/rpi3/crashes/20220307/ > > Hope this is of some use.... > > bob prohaska -- Dmitry Salychev --=-=-=--