panic with 4.10p4 and ipfw2

[Andrea Venturoli]

Hello.

A box of mine, which acts as firewall/bridge, is experiencing frequent 
panics.
As said in the subject line, it's a 4.10-RELEASE-p4 with ipfw2 enabled 
in the kernel.


I've run through post mortem kernel analisys and found out that the 
crashes are always related to ipfw2; specifically I get:
 > panic: free: multiple frees

Here is the complete backtrack:
> #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
> #1  0xc0150993 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
> #2  0xc0150db8 in poweroff_wait (junk=0xc02354ac, howto=-1071427665)
>     at /usr/src/sys/kern/kern_shutdown.c:595
> #3  0xc0208a3e in trap_fatal (frame=0xc023a3e4, eva=48)
>     at /usr/src/sys/i386/i386/trap.c:974
> #4  0xc0208711 in trap_pfault (frame=0xc023a3e4, usermode=0, eva=48)
>     at /usr/src/sys/i386/i386/trap.c:867
> #5  0xc02082fb in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 0,
>       tf_esi = 0, tf_ebp = -1071406036, tf_isp = -1071406064,
>       tf_ebx = -1071330820, tf_edx = 6864896, tf_ecx = -1054588914,
>       tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071892584, tf_cs = 8,
>       tf_eflags = 66182, tf_esp = -967647568, tf_ss = 0})
>     at /usr/src/sys/i386/i386/trap.c:466
> #6  0xc01c3798 in acquire_lock (lk=0xc024c9fc)
>     at /usr/src/sys/ufs/ffs/ffs_softdep.c:266
> #7  0xc01c8e7c in softdep_count_dependencies (bp=0xc652deb0, wantcount=0)
>     at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
> #8  0xc01cc0d8 in ffs_fsync (ap=0xc023a4a0)
>     at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
> #9  0xc01cabab in ffs_sync (mp=0xc123fc00, waitfor=2, cred=0xc0a3e800,
>     p=0xc026dbe0) at vnode_if.h:558
> #10 0xc0181737 in sync (p=0xc026dbe0, uap=0x0)
>     at /usr/src/sys/kern/vfs_syscalls.c:583
> #11 0xc015072e in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
> #12 0xc0150db8 in poweroff_wait (junk=0xc0218cff, howto=-1051816704)
>     at /usr/src/sys/kern/kern_shutdown.c:595
> #13 0xc014c41f in free (addr=0xc18fc100, type=0xc0249420)
>     at /usr/src/sys/kern/kern_malloc.c:385
> #14 0xc01a56ce in lookup_dyn_rule (pkt=0xc023a650, match_direction=0xc023a5c8,
>     tcp=0xc0b26b50) at /usr/src/sys/netinet/ip_fw2.c:784
> #15 0xc01a6ae7 in ipfw_chk (args=0xc023a630)
>     at /usr/src/sys/netinet/ip_fw2.c:1900
> #16 0xc01aa5f5 in ip_output (m0=0xc0b26b00, opt=0x0, ro=0xd0bfb0fc, flags=0,
>     imo=0x0, inp=0xd0bfb0c0) at /usr/src/sys/netinet/ip_output.c:733
> #17 0xc01afc51 in tcp_output (tp=0xd0bfb180)
>     at /usr/src/sys/netinet/tcp_output.c:953
> #18 0xc01ae977 in tcp_input (m=0xc0b26b00, off0=20, proto=6)
>     at /usr/src/sys/netinet/tcp_input.c:2229
> #19 0xc01a8f1c in ip_input (m=0xc0b26b00)
>     at /usr/src/sys/netinet/ip_input.c:934
> #20 0xc01a8f7b in ipintr () at /usr/src/sys/netinet/ip_input.c:955
> #21 0xc01fbd89 in swi_net_next ()
> #22 0xc0156a69 in softclock () at /usr/src/sys/kern/kern_timeout.c:131
> #23 0xc01fbd43 in doreti_swi ()



So, free is called from the following fragment:
> /**
> * lookup a dynamic rule.
>  */
> static ipfw_dyn_rule *
> lookup_dyn_rule(struct ipfw_flow_id *pkt, int *match_direction,
>         struct tcphdr *tcp)
> {
>        /*
>          * stateful ipfw extensions.
>          * Lookup into dynamic session queue
>          */
> #define MATCH_REVERSE   0
> #define MATCH_FORWARD   1
> #define MATCH_NONE      2
> #define MATCH_UNKNOWN   3
>         int i, dir = MATCH_NONE;
>         ipfw_dyn_rule *prev, *q=NULL;
> 
>         if (ipfw_dyn_v == NULL)
>                 goto done;      /* not found */
>         i = hash_packet( pkt );
>         for (prev=NULL, q = ipfw_dyn_v[i] ; q != NULL ; ) {
>                 if (q->dyn_type == O_LIMIT_PARENT && q->count)
>                         goto next;
>                 if (TIME_LEQ( q->expire, time_second)) { /* expire entry */
> =>                      UNLINK_DYN_RULE(prev, ipfw_dyn_v[i], q);
>                         continue;
>                 }
>                 if (pkt->proto == q->id.proto &&
>                     q->dyn_type != O_LIMIT_PARENT) {


I'm no kernel expert, so take my observation for what they might be 
worth, but:

> (kgdb) p *q
> $24 = {next = 0xc18a2d00, rule = 0xc6523b3c, parent = 0xd0001,
>   pcnt = 13916504069872025600, bcnt = 11709303859986432, id = {dst_ip = 0,
>     src_ip = 0, dst_port = 15744, src_port = 49469, proto = 164 '\244',
>     flags = 129 '\201'}, expire = 0, bucket = 4294967295, state = 4294967295,
>   ack_fwd = 0, ack_rev = 0, dyn_type = 0, count = 0}
> (kgdb)

These values do not make much sense to me... maybe the mess has already 
happened?

Any hint?
Is ipfw2 known to be broken in 4_10? Should I upgrade to 4_STABLE?
Or is it just a matter of finding a better configuration for all the 
relevant sysctl (which are all set to their default values)?

Really any help is appreciated!!!

  bye & Thanks
	av.