From nobody Fri Mar 18 18:02:37 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A9C621A2D7A2 for ; Fri, 18 Mar 2022 18:02:39 +0000 (UTC) (envelope-from mike@mail.karels.net) Received: from mail.karels.net (mail.karels.net [216.160.39.52]) by mx1.freebsd.org (Postfix) with ESMTP id 4KKsL64n3Qz4q9S for ; Fri, 18 Mar 2022 18:02:38 +0000 (UTC) (envelope-from mike@mail.karels.net) Received: from mail.karels.net (localhost [127.0.0.1]) by mail.karels.net (8.16.1/8.16.1) with ESMTPS id 22II2bOv024962 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Fri, 18 Mar 2022 13:02:37 -0500 (CDT) (envelope-from mike@mail.karels.net) Received: (from mike@localhost) by mail.karels.net (8.16.1/8.16.1/Submit) id 22II2bvI024961; Fri, 18 Mar 2022 13:02:37 -0500 (CDT) (envelope-from mike) Message-Id: <202203181802.22II2bvI024961@mail.karels.net> To: freebsd-net@freebsd.org From: Mike Karels Reply-to: mike@karels.net Subject: kernel epoch crash in IPv4 multicast code List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <24959.1647626557.1@mail.karels.net> Content-Transfer-Encoding: quoted-printable Date: Fri, 18 Mar 2022 13:02:37 -0500 X-Rspamd-Queue-Id: 4KKsL64n3Qz4q9S X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of mike@mail.karels.net has no SPF policy when checking 216.160.39.52) smtp.mailfrom=mike@mail.karels.net X-Spamd-Result: default: False [-1.61 / 15.00]; HAS_REPLYTO(0.00)[mike@karels.net]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.92)[-0.920]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_ADDR_EQ_FROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_NA(0.00)[karels.net]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net]; FORGED_SENDER(0.30)[mike@karels.net,mike@mail.karels.net]; RCVD_NO_TLS_LAST(0.10)[]; R_SPF_NA(0.00)[no SPF record]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:209, ipnet:216.160.36.0/22, country:US]; FROM_NEQ_ENVFROM(0.00)[mike@karels.net,mike@mail.karels.net] X-ThisMailContainsUnwantedMimeParts: N It looks like the IPv4 multicast code has not been fully converted to use epochs. I installed this week's snapshot of -current, configured and started mrouted, and started rwhod -m. The system crashed shortly thereafter with this: panic: Assertion in_epoch(net_epoch_preempt) failed at /usr/src/sys/netine= t/ip_output.c:343 cpuid =3D 15 time =3D 1647609865 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01b51a= 39d0 vpanic() at vpanic+0x17f/frame 0xfffffe01b51a3a20 panic() at panic+0x43/frame 0xfffffe01b51a3a80 ip_output() at ip_output+0x15f9/frame 0xfffffe01b51a3b80 phyint_send() at phyint_send+0x107/frame 0xfffffe01b51a3be0 ip_mdq() at ip_mdq+0x259/frame 0xfffffe01b51a3c60 X_ip_mrouter_set() at X_ip_mrouter_set+0x9e4/frame 0xfffffe01b51a3d30 sosetopt() at sosetopt+0xee/frame 0xfffffe01b51a3d80 kern_setsockopt() at kern_setsockopt+0xad/frame 0xfffffe01b51a3de0 sys_setsockopt() at sys_setsockopt+0x24/frame 0xfffffe01b51a3e00 amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe01b51a3f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01b51a3f30 --- syscall (105, FreeBSD ELF64, sys_setsockopt), rip =3D 0x821b72dda, rsp= =3D 0x8204c06f8, rbp =3D 0x8204c0750 --- KDB: enter: panic The kgdb backtrace is appended. It looks like ip_mroute is protected in the forwarding path (it's called from ip_input) and the output path, but not in the setup path from setsockopt(). At least the MRT_ADD_MFC call needs to enter an epoch. I tried adding epoch handling in add_mfc(), and that seems to work. The alternative would be to do it in Xip_mrouter_set() so it would cover all the calls. Any opinions? Mike (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3Dtextdump@entry=3D1) at /usr/src/sys/kern/kern_shutdown.c:406 #2 0xffffffff80c0b8ed in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:527 #3 0xffffffff80c0bdfe in vpanic ( fmt=3D0xffffffff8120c2ec "Assertion %s failed at %s:%d", ap=3D) at /usr/src/sys/kern/kern_shutdown.c:965 #4 0xffffffff80c0bb53 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:889 #5 0xffffffff80dde8f9 in ip_output (m=3Dm@entry=3D0xfffff802de8cb000, = opt=3Dopt@entry=3D0x0, ro=3Dro@entry=3D0x0, flags=3Dflags@entry=3D1, = imo=3Dimo@entry=3D0xfffffe01b51a3b98, inp=3Dinp@entry=3D0x0) at /usr/src/sys/netinet/ip_output.c:343 #6 0xffffffff82847317 in send_packet (vifp=3D0xfffff801c8620000, = m=3D) at /usr/src/sys/netinet/ip_mroute.c:1768 #7 phyint_send (ip=3Dip@entry=3D0xfffff8000ff1fc68, vifp=3D0xfffff801c862= 0000, = m=3D, m@entry=3D0xfffff8000ff1fc00) at /usr/src/sys/netinet/ip_mroute.c:1745 #8 0xffffffff828465c9 in ip_mdq (m=3D0xfffff8000ff1fc00, ifp=3D, = rt=3Drt@entry=3D0xfffff80025175b00, xmt_vif=3Dxmt_vif@entry=3D65535) at /usr/src/sys/netinet/ip_mroute.c:1653 #9 0xffffffff82844924 in add_mfc (mfccp=3D0xfffffe01b51a3ca0) at /usr/src/sys/netinet/ip_mroute.c:1168 #10 X_ip_mrouter_set (so=3D, = so@entry=3D, = sopt=3D, = sopt@entry=3D) at /usr/src/sys/netinet/ip_mroute.c:469 #11 0xffffffff80cbbafe in sosetopt (so=3D0xfffff8000fb1d3c0,