ipv6 panic in 6.0 ([kris@FreeBSD.org: kern/85780: 'panic: bogus refcnt 0' in routing/ipv6])

[Kris Kennaway]

I've been adding KTR debugging to try and track down the cause of this
recurring problem (FYI: debug.mpsafenet=0 is no longer working around
it).  To refresh your memory, here is the panic:

db> wh
Tracing pid 24 tid 100012 td 0xfffff802be9fa560
panic() at panic+0x164
rtfree() at rtfree+0xb4
nd6_na_output() at nd6_na_output+0x540
nd6_ns_input() at nd6_ns_input+0x738
icmp6_input() at icmp6_input+0xc38
ip6_input() at ip6_input+0x1038
netisr_processqueue() at netisr_processqueue+0x7c
swi_net() at swi_net+0xdc
ithread_execute_handlers() at ithread_execute_handlers+0x144
ithread_loop() at ithread_loop+0xa4
fork_exit() at fork_exit+0x94
fork_trampoline() at fork_trampoline+0x8
db>

It's always in nd6_na_output() although the trace beyond this point
varies.  However that doesn't tell us what leaked the reference count
prior to this stack trace.

So far I have narrowed it down to:

db> show ktr/v
                               Timestamp --v
    
9320 (0xfffff802be9fa560:cpu5) 1815572139270 net/route.c.247: Removing ref -> 0 0xfffff80227cefc20

^-- This is the cause of the panic in rtfree(), since it tries to decrement from 0.

9319 (0xfffff802be9fa560:cpu5) 1815572138338 netinet6/nd6_nbr.c.1028: Freeing route 0xfffff80227cefc20 with ref 0

^-- This is the call to rtfree() above, which is here at the end of
nd6_na_output():

        if (ro.ro_rt) {         /* we don't cache this route. */
                RTFREE(ro.ro_rt);
        }
        return;


9318 (0xfffff802be9fa560:cpu5) 1815572070306 net/route.c.247: Removing ref -> 1 0xfffff80227cefc20

This is the previous time rtfree() was run

9317 (0xfffff802be9fa560:cpu5) 1815572068930 netinet6/in6_src.c.703: rtfree 0xfffff80227cefc20

^-- this is the call to rtfree in 9318, which is at the end of
in6_selectif()

        if (rt && rt == sro.ro_rt)
                RTFREE(rt);
        return (0);

My next step is to add KTR logging to all the callers of
in6_selectif() to backtrace another level, but perhaps someone has
ideas what can be going wrong from the partial trace already.

9316 (0xfffff802be9fa560:cpu5) 1815572067244 net/route.c.198: Adding ref -> 0 0xfffff80227cefc20

This is in rtalloc1():

                } else {
                        KASSERT(rt == newrt, ("locking wrong route"));
                        RT_LOCK(newrt);
                        RT_ADDREF(newrt);

I suppose I need to also add KTR logging to the callers of rtalloc1().

9315 (0xfffff802be9fa560:cpu5) 1815572057262 netinet6/nd6.c.877: Removing ref -> 1 0xfffff80227cefc20

This is in nd6_lookup():

        }
        RT_LOCK_ASSERT(rt);
        RT_REMREF(rt);
        /*
         * Validation for the entry.
         * Note that the check for rt_llinfo is necessary because a cloned
         * route from a parent route that has the L flag (e.g. the default

NB: The RT_LOCK_ASSERT() is superfluous here since RT_REMREF() already
asserts it.

9314 (0xfffff802be9fa560:cpu5) 1815572046008 net/route.c.198: Adding ref -> 0 0xfffff80227cefc20

Kris

P.S. This comment in netinet6/ip6_output.c appears to be bogus, since
RTFREE is only a single statement:

        if (ro == &ip6route && ro->ro_rt) { /* brace necessary for RTFREE */
                RTFREE(ro->ro_rt);
        } else if (ro_pmtu == &ip6route && ro_pmtu->ro_rt) {
                RTFREE(ro_pmtu->ro_rt);
        }
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20060306/151f4784/attachment.bin