Panic in packet filter

Theodor-Iulian Ciobanu thciobanu at nth.ro
Thu Apr 12 22:29:40 UTC 2012 

On Thu, 12 Apr 2012 15:01:46 +0200
Ermal Luçi <eri at freebsd.org> wrote:

> Hello,
> 
> On Thu, Apr 12, 2012 at 1:16 PM, Theodor-Iulian Ciobanu
> <thciobanu at nth.ro> wrote:
> > Hello,
> >
> > I came across this same issue yesterday on a system I have just set
> > up. I'm currently using the default kernel:
> >
> > FreeBSD changeme 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3
> > 07:46:30 UTC 2012
> > root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
> >
> > with pf obviously loaded as a module. Even with kern.smp.disabled=1
> > pf will crash as soon as it matches a rule that contains tables with
> > counters (I added such a table with just three addresses).
> >
> > I'll have this machine around for testing for about a week or so
> > and am willing to try out any available patches to help fix the
> > issue.
> >
> 
> Try this patch
> http://people.freebsd.org/~eri/pf_table_counter_fix.diff. It should
> fix the issue for you.
> 
> Seems there is a forgotten pool initialization for this, my fault!
> 
> Though looking at it the whole thing seems a microoptimization that is
> still present on latest OpenBSD code,
> that saves about 16bytes!
> 
> Anyway see if it fixes the issue to get this committed.

Great use of 16b, as it doesn't seem to crash anymore, at least in a
simple synthetic test (uploading C:\Windows from 2 systems at once
through ftp, 10 transfer connections each).

Thank you!

> > On Fri Feb 24 14:47:53 2012
> > iskander at apple-park.kiev.ua (Alexander Vyrlanovich) wrote:
> >
> >>
> >> On 24 Feb 2012, at 11:10, Ali Mdidech wrote:
> >>
> >> > Hi Ermal,
> >> >
> >> > 2012/2/24 Ermal Lu?i <eri at freebsd.org>:
> >> >> On Thu, Feb 23, 2012 at 8:44 AM, Ali Mdidech <ali at moua7.com>
> >> >> wrote:
> >> >>> Hi List,
> >> >>>
> >> >>> I've a box that panics multiple times randomly since a year
> >> >>> whatever the release is (8 or 9)
> >> >>> The crash dump shows that the problem is related to pf.
> >> >>> Is this some sort of identified bug?
> >> >>> Below some info and my pf.conf file.
> >> >>>
> >> >>> Thank you very much for your help.
> >> >>>
> >> >>
> >> >> Can you try do disable SMP through sysctl and see if you still
> >> >> get this?
> >> >> What are you doing to get the panic?
> >> >
> >> > Well, I'm able now to avoid or reproduce the panic.
> >> > Disabling counters in <ssh_brute> table makes the server stable
> >> > enough and no panic for 48 hours.
> >> > Restoring the counters and adding a host in the table by hand
> >> > (pfctl -t ssh_brute -T add someip) provokes the panic within few
> >> > seconds. I've disabled smp (adding kern.smp.disabled=1 in
> >> > loader.conf and rebooting) => kernel still panics.
> >> >
> >> > FreeBSD somehost 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Sat Jan 21
> >> > 09:31:30 CET 2012     root at somehost:/usr/obj/usr/src/sys/DDX3KRNL
> >> > i386
> >> I can confirm that problem with counters in pf tables persist
> >> at last on i386 and amd64. My systems is:
> >>
> >> FreeBSD gw 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Tue Jan  3 15:55:41
> >> EET 2012
> >> root at gw:/usr/obj/usr/src/sys/GW3  amd64
> >>
> >> FreeBSD gw2 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Wed Jan 25 13:52:48
> >> EET 2012
> >> root at gw2:/usr/obj/usr/src/sys/GWS90  i386
> >>
> >> pf + altq compiled in kernel
> >>
> >> Same result: kernel panic. Without counters systems is rock solid.
> >>
> >> >> Also its very helpful to know the `uname -a` command output.
> >> >>
> >> >>> panic: page fault
> >> >>>
> >> >>> GNU gdb 6.1.1 [FreeBSD]
> >> >>> Copyright 2004 Free Software Foundation, Inc.
> >> >>> GDB is free software, covered by the GNU General Public
> >> >>> License, and you are
> >> >>> welcome to change it and/or distribute copies of it under
> >> >>> certain conditions.
> >> >>> Type "show copying" to see the conditions.
> >> >>> There is absolutely no warranty for GDB.  Type "show warranty"
> >> >>> for details.
> >> >>> This GDB was configured as "i386-marcel-freebsd"...
> >> >>>
> >> >>> Unread portion of the kernel message buffer:
> >> >>>
> >> >>>
> >> >>> Fatal trap 12: page fault while in kernel mode
> >> >>> cpuid = 0; apic id = 00
> >> >>> fault virtual address   = 0x6c
> >> >>> fault code              = supervisor read, page not present
> >> >>> instruction pointer     = 0x20:0xc0a25dc0
> >> >>> stack pointer           = 0x28:0xc4df5910
> >> >>> frame pointer           = 0x28:0xc4df5954
> >> >>> code segment            = base 0x0, limit 0xfffff, type 0x1b
> >> >>>                        = DPL 0, pres 1, def32 1, gran 1
> >> >>> processor eflags        = interrupt enabled, resume, IOPL = 0
> >> >>> current process         = 12 (irq256: em0:rx 0)
> >> >>> trap number             = 12
> >> >>> panic: page fault
> >> >>> cpuid = 0
> >> >>> KDB: stack backtrace:
> >> >>> #0 0xc08380b7 at kdb_backtrace+0x47
> >> >>> #1 0xc0805617 at panic+0x117
> >> >>> #2 0xc0aebcc3 at trap_fatal+0x323
> >> >>> #3 0xc0aec802 at trap+0x182
> >> >>> #4 0xc0ad5f8c at calltrap+0x6
> >> >>> #5 0xc589f7cc at pfr_update_stats+0x1cc
> >> >>> #6 0xc588de21 at pf_test+0x981
> >> >>> #7 0xc5895e79 at pf_check_in+0x39
> >> >>> #8 0xc08c3c68 at pfil_run_hooks+0x78
> >> >>> #9 0xc08e18ae at ip_input+0x24e
> >> >>> #10 0xc08c2d9f at netisr_dispatch_src+0x8f
> >> >>> #11 0xc08c3040 at netisr_dispatch+0x20
> >> >>> #12 0xc08b9721 at ether_demux+0x171
> >> >>> #13 0xc08b9b6f at ether_nh_input+0x37f
> >> >>> #14 0xc08c2d9f at netisr_dispatch_src+0x8f
> >> >>> #15 0xc08c3040 at netisr_dispatch+0x20
> >> >>> #16 0xc08b9269 at ether_input+0x19
> >> >>> #17 0xc05b383f at em_rxeof+0x30f
> >> >>> Uptime: 1h45m44s
> >> >>> Physical memory: 2002 MB
> >> >>> Dumping 185 MB: 170 154 138 122 106 90 74 58 42 26 10
> >> >>>
> >> >>> Reading symbols from /boot/kernel/pf.ko...Reading symbols from
> >> >>> /boot/kernel/pf.ko.symbols...
> >> >>> done.
> >> >>> done.
> >> >>> Loaded symbols for /boot/kernel/pf.ko
> >> >>> #0  doadump (textdump=1) at pcpu.h:244
> >> >>> 244     pcpu.h: No such file or directory.
> >> >>>        in pcpu.h
> >> >>> (kgdb) #0  doadump (textdump=1) at pcpu.h:244
> >> >>> #1  0xc08053ba in kern_reboot (howto=260)
> >> >>>    at /usr/src/sys/kern/kern_shutdown.c:442
> >> >>> #2  0xc0805651 in panic (fmt=Variable "fmt" is not available.
> >> >>> ) at /usr/src/sys/kern/kern_shutdown.c:607
> >> >>> #3  0xc0aebcc3 in trap_fatal (frame=0xc4df58d0, eva=108)
> >> >>>    at /usr/src/sys/i386/i386/trap.c:975
> >> >>> #4  0xc0aec802 in trap (frame=0xc4df58d0) at /usr/src/sys/i386/
> >> >>> i386/trap.c:352
> >> >>> #5  0xc0ad5f8c in calltrap () at /usr/src/sys/i386/i386/
> >> >>> exception.s:168
> >> >>> #6  0xc0a25dc0 in uma_zalloc_arg (zone=0x0, udata=0x0,
> >> >>> flags=257) at pcpu.h:244
> >> >>> #7  0xc589f7cc in pfr_update_stats (kt=0xc58d44d8,
> >> >>> a=0xc56aa01a, af=2 '\002',
> >> >>>    len=52, dir_out=0, op_pass=0, notrule=0) at uma.h:305
> >> >>> #8  0xc588de21 in pf_test (dir=1, ifp=0xc5253c00,
> >> >>> m0=0xc4df5acc, eh=0x0,
> >> >>>    inp=0x0)
> >> >>> at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c: 7057
> >> >>> #9  0xc5895e79 in pf_check_in (arg=0x0, m=0xc4df5acc,
> >> >>> ifp=0xc5253c00, dir=1,
> >> >>>    inp=0x0) at /usr/src/sys/modules/pf/../../contrib/pf/net/
> >> >>> pf_ioctl.c:4139
> >> >>> #10 0xc08c3c68 in pfil_run_hooks (ph=0xc0d685e0, mp=0xc4df5b24,
> >> >>>    ifp=0xc5253c00, dir=1, inp=0x0)
> >> >>> at /usr/src/sys/net/pfil.c:82 #11 0xc08e18ae in ip_input
> >> >>> (m=0xc567db00) at /usr/src/sys/netinet/ip_input.c:510
> >> >>> #12 0xc08c2d9f in netisr_dispatch_src (proto=1, source=0,
> >> >>> m=0xc567db00)
> >> >>>    at /usr/src/sys/net/netisr.c:1013
> >> >>> #13 0xc08c3040 in netisr_dispatch (proto=1, m=0xc567db00)
> >> >>>    at /usr/src/sys/net/netisr.c:1104
> >> >>> #14 0xc08b9721 in ether_demux (ifp=0xc5253c00, m=0xc567db00)
> >> >>>    at /usr/src/sys/net/if_ethersubr.c:937
> >> >>> #15 0xc08b9b6f in ether_nh_input (m=0xc567db00)
> >> >>>    at /usr/src/sys/net/if_ethersubr.c:756
> >> >>> #16 0xc08c2d9f in netisr_dispatch_src (proto=9, source=0,
> >> >>> m=0xc567db00)
> >> >>>    at /usr/src/sys/net/netisr.c:1013
> >> >>> #17 0xc08c3040 in netisr_dispatch (proto=9, m=0xc567db00)
> >> >>>    at /usr/src/sys/net/netisr.c:1104
> >> >>> #18 0xc08b9269 in ether_input (ifp=0xc5253c00, m=0xc567db00)
> >> >>>    at /usr/src/sys/net/if_ethersubr.c:797
> >> >>> #19 0xc05b383f in em_rxeof (rxr=0xc520bc00, count=99, done=0x0)
> >> >>>    at /usr/src/sys/dev/e1000/if_em.c:4340
> >> >>> #20 0xc05b3a06 in em_msix_rx (arg=0xc520bc00)
> >> >>>    at /usr/src/sys/dev/e1000/if_em.c:1577
> >> >>> #21 0xc07da6eb in intr_event_execute_handlers (p=0xc5157588,
> >> >>> ie=0xc5241680)
> >> >>>    at /usr/src/sys/kern/kern_intr.c:1257
> >> >>> #22 0xc07dbeaa in ithread_loop (arg=0xc52506e0)
> >> >>>    at /usr/src/sys/kern/kern_intr.c:1270
> >> >>> #23 0xc07d78f7 in fork_exit (callout=0xc07dbe30 <ithread_loop>,
> >> >>>    arg=0xc52506e0, frame=0xc4df5d28) at /usr/src/sys/kern/
> >> >>> kern_fork.c:995
> >> >>> #24 0xc0ad6004 in fork_trampoline () at /usr/src/sys/i386/i386/
> >> >>> exception.s:275
> >> >>> (kgdb)
> >> >>>
> >> >>>
> >> >>> ################## pf.conf ##################
> >> >>> ext_if = "em0"
> >> >>>
> >> >>> public_tcp_ports = "{21,25,53,80,143,443,873,993,50021:50121}"
> >> >>> public_udp_ports = "53"
> >> >>>
> >> >>> table <secure> {someip}
> >> >>> table <ssh_brute> persist counters
> >> >>>
> >> >>> ### Redirection for SMTP
> >> >>> rdr on $ext_if proto tcp from any to $ext_if port 225 ->
> >> >>> $ext_if port 25
> >> >>>
> >> >>> ### Block everything in an pass everything out
> >> >>> pass out on $ext_if all modulate state
> >> >>> block in on $ext_if all
> >> >>>
> >> >>> ### secure users
> >> >>> pass in quick on $ext_if proto tcp from <secure> to any flags
> >> >>> S/SA \ modulate state
> >> >>>
> >> >>> ### public tcp/udp ports rules
> >> >>> pass in on $ext_if proto udp to $ext_if port $public_udp_ports
> >> >>> pass in on $ext_if proto tcp to $ext_if port $public_tcp_ports
> >> >>> flags S/SA \
> >> >>> modulate state
> >> >>>
> >> >>> ### block ssh bruteforce
> >> >>> block in quick from <ssh_brute>
> >> >>> pass in quick on $ext_if proto tcp to $ext_if port 22 flags
> >> >>> S/SA modulate state \
> >> >>> (max-src-conn 5, max-src-conn-rate 10/60, overload <ssh_brute>
> >> >>> flush global)
> >> >>>
> >> >>> ### block icmp timestamp request/response
> >> >>> block in quick on $ext_if inet proto icmp all icmp-type {13,
> >> >>> 14} pass in quick on $ext_if proto icmp all
> >> >>>
> >> >>> ############ end pf.conf ##############
> >> >>>
> >> >>> --
> >> >>> Ali Mdidech
> >> >>> _______________________________________________
> >> >>> freebsd-pf at freebsd.org mailing list
> >> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> >> >>> To unsubscribe, send any mail to "freebsd-pf-
> >> >>> unsubscribe at freebsd.org"
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Ermal
> >> >
> >> > --
> >> > Ali Mdidech
> >> > _______________________________________________
> >> > freebsd-pf at freebsd.org mailing list
> >> > http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> >> > To unsubscribe, send any mail to
> >> > "freebsd-pf-unsubscribe at freebsd.org"
> >>
> >> ????????? ??????????
> >> --------------------------
> >> ????????? ?????????????
> >> ??? "???"
> >
> > --
> > Theo
> > _______________________________________________
> > freebsd-pf at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> > To unsubscribe, send any mail to
> > "freebsd-pf-unsubscribe at freebsd.org"

-- 
Theo

More information about the freebsd-pf mailing list