From nobody Mon Mar 21 12:41:15 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 778691A2E425 for ; Mon, 21 Mar 2022 12:41:22 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KMZ4232ftz3rl2; Mon, 21 Mar 2022 12:41:22 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647866482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kxDFIpRudpYlC+6Tb++99XfP5Bx9nL9ziiErf/qDhRQ=; b=mfLTm60TN7UQsVPfEm85RXie3tjLQmiRCQAuoTR7XHd9k7q1K93ONA2Au8qOsuE4R5+QnH DWxVkJ3/AHk8hi4PfGyedPH3vPE0azO9Kjvaxkn2A6Q9x4Dk9yreYgkEpsVmkkGUSdQFsB oBEfFHG/9ZW7ZqXAKJUECBbuZsERAWHaQZ5TbUvHKX7tnLNQJhbc91NcbAL9QjDKBjILQz 3QjxoaOCzMQGvu6+8m7VG4JgJVXEvr4V3G5yo5dw15arzufsi82bFuvrso/VkNfLu7qFT5 OgyZe7b3vKqX3uFSQPSfFlhl7cuek9CA0+ccNw/f/7KzmTWcftg3qXLs9RUMYQ== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 316EE2A880; Mon, 21 Mar 2022 12:41:22 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 3A9E72EDD1; Mon, 21 Mar 2022 13:41:20 +0100 (CET) From: Kristof Provost To: Mike Karels Cc: freebsd-net@freebsd.org Subject: Re: kernel epoch crash in IPv4 multicast code Date: Mon, 21 Mar 2022 13:41:15 +0100 X-Mailer: MailMate (1.14r5852) Message-ID: <9E6CA0F5-5E02-4458-8D9F-C7F8F1715BFC@FreeBSD.org> In-Reply-To: <202203181802.22II2bvI024961@mail.karels.net> References: <202203181802.22II2bvI024961@mail.karels.net> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647866482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kxDFIpRudpYlC+6Tb++99XfP5Bx9nL9ziiErf/qDhRQ=; b=ofAi153Ewh/Ij6VuxNfZs7PAX2mWnmzOIOM0Pk5G6oV4WeJSFPKC9xXxeP46aM1jO+q++l rHEC4s3Upbc/m1cwJuFlWtWS8txHfc9byz8kdOWDek+WgmEZrf3E89kt/9Jn6LdWgU5F13 ewW6GUBv6Fl0UDvxTVpFNn/4gcFI/NFYiEPKK1ykEdEv+wjWcNkg4934dXt27w9EMEoGrD hozQLVd8eyxCcD2DseUxtaz+NtdGspbexQfbZIDKC6QbGJjLA/IrTCxI4RFntStd/byCnO NsIkCQy4gV4CEMV24EjSVWZJdjsUQd5WLB+rpiBLrAYl5pg/lyNqxij7cPtBoA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647866482; a=rsa-sha256; cv=none; b=QIVRCeOsZByrMoh6nEEXsH+oWRXjgnfdPn3pzj79DKUB1AlSX+kgIO7pQY97z9LBlo0c2R xXDQbphtG4RliY8lK2rbWecwv+DwmBRfUApGOREULlLg5H3fnDQ49q5cBaFXfaSLQ/ka4V w7TbDYZGHiXRTe3FI3QDw2P+ExVyJWacWDw0vs/7TrQfR01jzAXfm8r0Kbf4Lh5ynEmkig 2wN2wtXZ+fUCzByM1P/MGpTlOooNlrWlqwAGTBUx/mLstxUy7KH8JZDgUX+mpnz35Cj9Yx UIm6sPT383t7qeS/Uos23qUEF/RVKK9JWb8soEkahz4RhLUjCUKSGrrTOhTzNQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 18 Mar 2022, at 19:02, Mike Karels wrote: > It looks like the IPv4 multicast code has not been fully converted to > use epochs. I installed this week's snapshot of -current, configured > and started mrouted, and started rwhod -m. The system crashed shortly > thereafter with this: > > panic: Assertion in_epoch(net_epoch_preempt) failed at /usr/src/sys/net= inet/ip_output.c:343 > cpuid =3D 15 > time =3D 1647609865 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01b= 51a39d0 > vpanic() at vpanic+0x17f/frame 0xfffffe01b51a3a20 > panic() at panic+0x43/frame 0xfffffe01b51a3a80 > ip_output() at ip_output+0x15f9/frame 0xfffffe01b51a3b80 > phyint_send() at phyint_send+0x107/frame 0xfffffe01b51a3be0 > ip_mdq() at ip_mdq+0x259/frame 0xfffffe01b51a3c60 > X_ip_mrouter_set() at X_ip_mrouter_set+0x9e4/frame 0xfffffe01b51a3d30 > sosetopt() at sosetopt+0xee/frame 0xfffffe01b51a3d80 > kern_setsockopt() at kern_setsockopt+0xad/frame 0xfffffe01b51a3de0 > sys_setsockopt() at sys_setsockopt+0x24/frame 0xfffffe01b51a3e00 > amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe01b51a3f30 > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01b51a3= f30 > --- syscall (105, FreeBSD ELF64, sys_setsockopt), rip =3D 0x821b72dda, = rsp =3D 0x8204c06f8, rbp =3D 0x8204c0750 --- > KDB: enter: panic > > The kgdb backtrace is appended. > > It looks like ip_mroute is protected in the forwarding path (it's calle= d > from ip_input) and the output path, but not in the setup path from > setsockopt(). At least the MRT_ADD_MFC call needs to enter an epoch. > I tried adding epoch handling in add_mfc(), and that seems to work. > The alternative would be to do it in Xip_mrouter_set() so it would cove= r > all the calls. Any opinions? > Your analysis looks reasonable. I think I=E2=80=99d suggest adding the NET_EPOCH_ENTER() calls in add_mfc= (). We already do that in add_vif(), so we=E2=80=99d be following existin= g choices. I=E2=80=99d also suggest adding NET_EPOCH_ASSERT() to everything which di= rectly or indirectly calls ip_output(). That should help us catch other p= otential issues like this one. Br, Kristof