Re: CURRENT: kernel panic in IPFW while stopping jails
- In reply to: Alastair Hogge : "Re: CURRENT: kernel panic in IPFW while stopping jails"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 28 Dec 2025 12:33:33 UTC
> On Dec 28, 2025, at 3:36 AM, Alastair Hogge <agh@riseup.net> wrote: ...sending from the right account this time... (Mail clients break becausse they only see the list-address in the To: header so don't use the right role). Removing my final "log" rule fixes this for me, yes. (That probably should be sufficient for anyone else to reproduce, I posted earlier in thread, but my final rule (before the automatic ones at 65535) is "reset log ip from any to any"). My panic output looks like this (there might be slight character misses because this is text copy/pasted from a screenshot (so the console zero font gets grabbed as Ø) I don't have good knowledge of how to drive the kernel debugger. Feel free to email me privately on-list if you can tell me a command I didn't want to reopen the other bug, but it might belong there. -Dan panic: Assertion tap != NULL failed at /usr/src/sys/netpfil/ipfw/ip_fw_bpf.c:127 cpuid = 9 tie = 1766922462 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+@x2b/frame Øxfffffe0123Øa35cØ vpanic() at vpanic+0x136/frame Øxfffffe01230a36f0 panic() at panic+0x43/frame Øxfffffe01230a3750 ipfw_bpf_tap() at ipfw_bpf_tap+@x104/frame Øxfffffe01230a3760 ipfw_chk() at ipfw_chk+@x1e37/frame Øxfffffe01230a3990 ipfw_check_packet() at ipfw_check_packet+@xf6/frame 0xfffffe01230a3a80 pfil_mbuf_in() at pfil_mbuf_in+@x58/frame @xfffffe01230a3ab0 ip input ) at ip_input+@x3f9/frame Bxfffffe01230a3b10 netisr_dispatch_src() at netisr_dispatch_src+@xb4/frame ØxfffffeØ123Øa3b7Ø ether_demux() at ether_demux+0x16a/frame Øxfffffe01230a3baØ ether_nh_input() at ether_nh_input+@x3ce/frame Øxfffffe01230a3bf0 _src+0xb4/frame netisr_dispatch_src() at netisr_dispatch_src+Øxb4/frame Øxfffffe0123Øa3c50 ether_input () at ether_input+@xd5/frame @xfffffe01230a3cb0 tcp_lro_flush_all() at top_lro_flush_all+@xdc/frame Øxfffffe01230a3d00 iflib_rxeof() at iflib_rxeof+@xbef/frame Øxfffffe01230a3e00 _task_in_rx() at _task _fn_rx+@x7a/frame Øxfffffe01230a3e40 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x18e/frame Øxfffffe01230a3ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+@xd3/frame @xfffffe01230a3ef0 _1oop+@xd3/frame fork_exit() at fork_exit+0x82/frame Øxfffffe01230a3f30 fork_trampoline() at fork_trampoline+@xe/frame Bxfffffe01230a3f30 But I don't actually get a kernel dump. lb> bt Tracing pid 8 tid 100032 td Bxfffff80004725000 kdb_enter() at kdb_enter+Øx33/frame ØxfffffeØ123Øa36f0 panic() at panic+0x43/frame Øxfffffe01230a3750 ipfw_bpf_tap() at ipfw_bpf_tap+@x184/frame ØxfffffeØ123Øa3760 ipfw_chk() at ipfw_chk+@x1e37/frame Øxfffffe01230a3990 ipfw_check_packet() at ipfw_check_packet+Øxf6/frame ØxfffffeØ123Øa3a8Ø pfil_mbuf _in() at pfil_mbuf_in+@x58/frame BxfffffeB1230a3ab0 ip_input() at ip_input+@x3f9/frame Øxfffffe01230a3b10 netisr_dispatch_src() at netisr_dispatch_src+@xb4/frame Øxfffffe01230a3b70 ether_demux() at ether_deмux+@x16a/frame Bxfffffe01230a3baø ether_nh_input() at ether_nh_input+Øx3ce/frame Bxfffffe01230a3bf8 netisr_dispatch_src() at netisr _dispatch_src+@xb4/fraмe Øxfffffe0123Øa3c5Ø ether_input() at ether_input+0xd5/frame Bxfffffe0123Øa3cbØ top_lro_flush_alll) at top_lro_flush_all+@xdc/frame @xfffffe01230a3dB0 iflib_rxeof() at iflib_rxeof+@xbef/frame Øxfffffe01230a3eB8 _task_fn_rx() at _task_fn_rx+@x7a/frame Øxfffffe0123Øa3e40 gtaskqueue_run_locked() at gtaskqueue_run_locked+@x18e/frame Øxfffffe01230a3ecØ gtaskqueue_thread_loop() at gtaskqueue_thread _1oop+Øxd3/frame Øxfffffe0123Øa3efE fork_exit() at fork_exit+0x82/fraмe Øxfffffe0123Øa3f30 fork_trampoline() at fork_trampoline+@xe/frame Bxfffffe0123Øa3f30 > > On 2025-12-26 17:32, A FreeBSD User wrote: >> Am Tage des Herren Thu, 25 Dec 2025 19:08:36 +0100 >> FreeBSD User <freebsd@walstatt-de.de> schrieb: >> >>> On Thu, 25 Dec 2025 18:30:45 +0100 (CET) >>> Ronald Klop <ronald-lists@klop.ws> wrote: >>> >>>> Do you use bpf or tap in your ipfw rules? >>>> A panic with that was mentioned on the 20th. And fixed in the mean time of I >>>> remember correctly. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291854 >>>> Regards,Ronald >>> >>> Indeed, all boxes in question do have a tap0 at least defined -but in only one >>> case used. >>> >>> Kind regards, >>> >>> oh >>> >>> >>>> >>>> Van: FreeBSD User <freebsd@walstatt-de.de> >>>> Datum: 25 december 2025 17:09 >>>> Aan: FreeBSD CURRENT <freebsd-current@freebsd.org> >>>> Onderwerp: CURRENT: kernel panic in IPFW while stopping jails >>>> >>>>> >>>>> >>>>> Hello, >>>>> >>>>> on recent CURRENT ipfw (in my case compiled into kernel) either restarting >>>>> "ipfw" via "service ipfw restart" or restarting jails using also ipfw on a >>>>> host also using ipfw (jail-hoster also ipfw compiled into kernel) causes a >>>>> fatal kernel crash. >>>>> >>>>> This issue is present since last week an wreak havok to several boxes with >>>>> OS installed on UFS/FFS SSDs. In one case I have only pictures/screenshots >>>>> made via smartphone - while crashing, kernel debugger input pops up on >>>>> console, but I'm able to typein something within the first seconds and this >>>>> is mostly "reboot" but gets stuck with "re" in most cases. "bt" freezes >>>>> system immediately. >>>>> >>>>> At least I can reproduce this misbehaviour on all recent CURRENT were IPFW >>>>> is compiled into kernel. Anybody else havong this trouble? >>>>> >>>>> Kind regards, >>>>> >>>>> Oliver >>>>> >>>>> Merry Christmas to everyone. >>>>> >>>>> -- >>>>> >>>>> A FreeBSD user >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> >> >> tap0 disabled/deleted. Same issue on every box. > > $ git bisect log > git bisect start > # status: waiting for both good and bad commits > # bad: [086bedb11a853801e82234b8a1a64f0df52d9e52] tools.build: also add > sys/_visible.h to SYSINCS > git bisect bad 086bedb11a853801e82234b8a1a64f0df52d9e52 > # status: waiting for good commit(s), bad commit known > # good: [44cb1e857f048d2326bdc1a032ccd2c04d2bcdc9] tcp: improve > credential handling in syncache > git bisect good 44cb1e857f048d2326bdc1a032ccd2c04d2bcdc9 > # good: [b0c7eaf83d21bbc333e247ab9e136965b3ca54ed] bhyve/slirp: Drop > privileges before entering capability mode > git bisect good b0c7eaf83d21bbc333e247ab9e136965b3ca54ed > # good: [6a75e3951506c12b42428a47710d07cadcdd723e] ofed/libibverbs: > remove strdupa() hack from config.h > git bisect good 6a75e3951506c12b42428a47710d07cadcdd723e > # bad: [1fad49baf390cb52f238e6c352d0bc0893c008c3] sdhci: Try to complete > the last transaction if dumping > git bisect bad 1fad49baf390cb52f238e6c352d0bc0893c008c3 > # good: [9d9974457ce8c6cf9023884ab457d4712dcc237f] bhyvectl: fix build > without BHYVE_SNAPSHOT > git bisect good 9d9974457ce8c6cf9023884ab457d4712dcc237f > # bad: [52395203f9ac40d321ed55d93e9887300261d3bf] MFV: Import blocklist > 2025-12-15 (8a4b011) > git bisect bad 52395203f9ac40d321ed55d93e9887300261d3bf > # good: [c112ad75605ccdfcb8bbce2f57b0e7a077f057f8] options: describe > WITH_IPFILTER_IPFS > git bisect good c112ad75605ccdfcb8bbce2f57b0e7a077f057f8 > # good: [8774a990ee4094f16d596d4b78e0f3239e5d0c88] bpf: modularize > ifnet(9) part of bpf > git bisect good 8774a990ee4094f16d596d4b78e0f3239e5d0c88 > # bad: [1615eff94cda8619561b73186ec8098cc8b14c5c] usb: don't create > ifnet(9) for usbus devices > git bisect bad 1615eff94cda8619561b73186ec8098cc8b14c5c > # good: [ddf4f9eda9c295082f17e7f26963666b72c97bb9] ipfw: create "ipfw0" > and "ipfwlog0" bpf tapping points without ifnet(9) > git bisect good ddf4f9eda9c295082f17e7f26963666b72c97bb9 > # bad: [3daae1ac1d82ecdcd855101bab5206e914b12350] ipfw: create a bpf tap > point for every log rule > git bisect bad 3daae1ac1d82ecdcd855101bab5206e914b12350 > # good: [1c5021f5251b231b614ad9cd175bcb4250495c12] ifconfig: print > warning and return success on ipfw0, ipfwlog0 cloning > git bisect good 1c5021f5251b231b614ad9cd175bcb4250495c12 > # first bad commit: [3daae1ac1d82ecdcd855101bab5206e914b12350] ipfw: > create a bpf tap point for every log rule > > https://codeberg.org/FreeBSD/freebsd-src/commit/3daae1ac1d82ecdcd855101bab5206e914b12350 > ipfw: create a bpf tap point for every log rule > > Dynamically allocate bpf tap points for every rule that has "log". > The name is "ipfw%u", where %u is substituted to the rule number. > The default catch all "ipfw0" tap still exists for compatibility > and it will catch packets in case if there are no bpf listeners > on a per-rule tap. > > Reviewed by: ae > Differential Revision: https://reviews.freebsd.org/D53877