kernel epoch crash in IPv4 multicast code

From: Mike Karels <mike_at_karels.net>
Date: Fri, 18 Mar 2022 18:02:37 UTC
It looks like the IPv4 multicast code has not been fully converted to
use epochs.  I installed this week's snapshot of -current, configured
and started mrouted, and started rwhod -m.  The system crashed shortly
thereafter with this:

panic: Assertion in_epoch(net_epoch_preempt) failed at /usr/src/sys/netinet/ip_output.c:343
cpuid = 15
time = 1647609865
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01b51a39d0
vpanic() at vpanic+0x17f/frame 0xfffffe01b51a3a20
panic() at panic+0x43/frame 0xfffffe01b51a3a80
ip_output() at ip_output+0x15f9/frame 0xfffffe01b51a3b80
phyint_send() at phyint_send+0x107/frame 0xfffffe01b51a3be0
ip_mdq() at ip_mdq+0x259/frame 0xfffffe01b51a3c60
X_ip_mrouter_set() at X_ip_mrouter_set+0x9e4/frame 0xfffffe01b51a3d30
sosetopt() at sosetopt+0xee/frame 0xfffffe01b51a3d80
kern_setsockopt() at kern_setsockopt+0xad/frame 0xfffffe01b51a3de0
sys_setsockopt() at sys_setsockopt+0x24/frame 0xfffffe01b51a3e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe01b51a3f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01b51a3f30
--- syscall (105, FreeBSD ELF64, sys_setsockopt), rip = 0x821b72dda, rsp = 0x8204c06f8, rbp = 0x8204c0750 ---
KDB: enter: panic

The kgdb backtrace is appended.

It looks like ip_mroute is protected in the forwarding path (it's called
from ip_input) and the output path, but not in the setup path from
setsockopt().  At least the MRT_ADD_MFC call needs to enter an epoch.
I tried adding epoch handling in add_mfc(), and that seems to work.
The alternative would be to do it in Xip_mrouter_set() so it would cover
all the calls.  Any opinions?

		Mike

(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=textdump@entry=1)
    at /usr/src/sys/kern/kern_shutdown.c:406
#2  0xffffffff80c0b8ed in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:527
#3  0xffffffff80c0bdfe in vpanic (
    fmt=0xffffffff8120c2ec "Assertion %s failed at %s:%d", ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:965
#4  0xffffffff80c0bb53 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:889
#5  0xffffffff80dde8f9 in ip_output (m=m@entry=0xfffff802de8cb000, 
    opt=opt@entry=0x0, ro=ro@entry=0x0, flags=flags@entry=1, 
    imo=imo@entry=0xfffffe01b51a3b98, inp=inp@entry=0x0)
    at /usr/src/sys/netinet/ip_output.c:343
#6  0xffffffff82847317 in send_packet (vifp=0xfffff801c8620000, 
    m=<optimized out>) at /usr/src/sys/netinet/ip_mroute.c:1768
#7  phyint_send (ip=ip@entry=0xfffff8000ff1fc68, vifp=0xfffff801c8620000, 
    m=<optimized out>, m@entry=0xfffff8000ff1fc00)
    at /usr/src/sys/netinet/ip_mroute.c:1745
#8  0xffffffff828465c9 in ip_mdq (m=0xfffff8000ff1fc00, ifp=<optimized out>, 
    rt=rt@entry=0xfffff80025175b00, xmt_vif=xmt_vif@entry=65535)
    at /usr/src/sys/netinet/ip_mroute.c:1653
#9  0xffffffff82844924 in add_mfc (mfccp=0xfffffe01b51a3ca0)
    at /usr/src/sys/netinet/ip_mroute.c:1168
#10 X_ip_mrouter_set (so=<optimized out>, 
    so@entry=<error reading variable: value is not available>, 
    sopt=<optimized out>, 
    sopt@entry=<error reading variable: value is not available>)
    at /usr/src/sys/netinet/ip_mroute.c:469
#11 0xffffffff80cbbafe in sosetopt (so=0xfffff8000fb1d3c0,