Kernel Panic on 9.0 and 9.1 with carp on BCE network interface

Tue Sep 25 13:32:30 UTC 2012

Hi,  Is there anything else I can provide you? the core and kernel.debug?
The panic is hapening once or twice a day on this server.

Thank you again for your help

> Date: Fri, 7 Sep 2012 17:14:41 +0400
> From: glebius at FreeBSD.org
> To: jl.dupont at outlook.com
> CC: freebsd-stable at FreeBSD.org
> Subject: Re: Kernel Panic on 9.0 and 9.1 with carp on BCE network interface
> 
> On Thu, Aug 30, 2012 at 02:39:10PM +0000, Jean-Luc Dupont wrote:
> J> Sorry, it seems that I didn't put the right backtrace :
> J> 
> J> #0  doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> 271                     dumpsys(&dumper);
> J> (kgdb) #0  doadump (textdump=Variable "textdump" is not available.
> J> ) at /usr/src/sys/kern/kern_shutdown.c:271
> J> #1  0xffffffff807fdf02 in kern_reboot (howto=260)
> J>     at /usr/src/sys/kern/kern_shutdown.c:448
> J> #2  0xffffffff807fe3e3 in panic (fmt=0x104 <Address 0x104 out of bounds>)
> J>     at /usr/src/sys/kern/kern_shutdown.c:636
> J> #3  0xffffffff80ad2700 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
> J> )
> J>     at /usr/src/sys/amd64/amd64/trap.c:857
> J> #4  0xffffffff80ad2a3d in trap_pfault (frame=0xffffff82e97a3500, usermode=0)
> J>     at /usr/src/sys/amd64/amd64/trap.c:773
> J> #5  0xffffffff80ad305e in trap (frame=0xffffff82e97a3500)
> J>     at /usr/src/sys/amd64/amd64/trap.c:456
> J> #6  0xffffffff80abd67f in calltrap ()
> J>     at /usr/src/sys/amd64/amd64/exception.S:228
> J> #7  0xffffffff8085f597 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
> J>     at /usr/src/sys/kern/uipc_mbuf.c:542
> J> #8  0xffffffff8092f2c8 in ip_fragment (ip=0xfffffe00970e0580, 
> J>     m_frag=0xffffff82e97a3728, mtu=Variable "mtu" is not available.
> J> ) at /usr/src/sys/netinet/ip_output.c:822
> J> #9  0xffffffff8092fc17 in ip_output (m=0xfffffe00970e0500, opt=Variable "opt" is not available.
> J> )
> J>     at /usr/src/sys/netinet/ip_output.c:653
> J> #10 0xffffffff80928713 in ip_forward (m=0xfffffe00970e0500, srcrt=Variable "srcrt" is not available.
> J> )
> J>     at /usr/src/sys/netinet/ip_input.c:1494
> J> #11 0xffffffff80929dc8 in ip_input (m=0xfffffe00970e0500)
> J>     at /usr/src/sys/netinet/ip_input.c:702
> 
> I don't see that this is CARP related. Do you use any firewall: pf or ipfw?
> 
> Can you please show the below session in gdb with discussed core file:
> 
> gdb> fr 9
> gdb> p mtu
> gdb> fr 7
> gdb> p off
> gdb> fr 8
> gdb> p m0
> gdb> p *m0
> 
> -- 
> Totus tuus, Glebius.
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

Hi,  

  Thank you very much for your reply, we are using IPFW with several VLAN and several CARP on intel igb and bce network cards on a dell poweredge servers.
When we stopped using the bce and using only the igb (with more vlans per interface) we don't have any more panics.

Here is the output of the debugger as asked :

(kgdb) fr 9
#9  0xffffffff8092fc17 in ip_output (m=0xfffffe00941c8300, opt=Variable "opt" is not available.
) at /usr/src/sys/netinet/ip_output.c:653
653             error = ip_fragment(ip, &m, mtu, ifp->if_hwassist, sw_csum);
(kgdb) p mtu
$1 = 1500
(kgdb) fr 7
#7  0xffffffff8085f597 in m_copym (m=0x0, off0=1500, len=1317, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542
542                     if (off < m->m_len)
(kgdb) p off
$2 = 1233
(kgdb) fr 8
#8  0xffffffff8092f2c8 in ip_fragment (ip=0xfffffe00941c8380, m_frag=0xffffff834869e7f8, mtu=Variable "mtu" is not available.
)
    at /usr/src/sys/netinet/ip_output.c:822
822                     m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) p m0
$3 = (struct mbuf *) 0xfffffe00941c8300
(kgdb) p *m0
$4 = {m_hdr = {mh_next = 0xfffffe0081d51800, mh_nextpkt = 0x0, mh_data = 0xfffffe00941c8380 "E", mh_len = 40, 
    mh_flags = 2, mh_type = 1, pad = "\000\000\000\000\000"}, M_dat = {MH = {MH_pkthdr = {
        rcvif = 0xfffffe0003b53800, header = 0x0, len = 267, flowid = 0, csum_flags = 0, csum_data = 65535, 
        tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0x0}}, MH_dat = {MH_ext = {
          ext_buf = 0x400092ae00400045 <Address 0x400092ae00400045 out of bounds>, ext_free = 0x16207, 
          ext_arg1 = 0x42ffff011d0000, ext_arg2 = 0x601005e00000000, ext_size = 2660147200, 
          ref_cnt = 0x40f7e20b010045, ext_type = -843971023}, 
        MH_databuf = "E\000@\000�\222\000@\ab\001\000\000\000\000\000\000\000\035\001��B\000\000\000\000\000^\000\001\006\000�\216\236�\200\b\000E\000\001\v��@\0001\006��H\025T�\n\n\vK\000\025��^h���\223R>\200\030\000r\213�\000\000\001\001\b\n�$*\200:��\a", '\0' <repeats 75 times>}}, 
    M_databuf = "\0008�\003\000���\000\000\000\000\000\000\000\000\v\001\000\000\000\000\000\000\000\000\000\000��", '\0' <repeats 18 times>, "E\000@\000�\222\000@\ab\001\000\000\000\000\000\000\000\035\001��B\000\000\000\000\000^\000\001\006\000�\216\236�\200\b\000E\000\001\v��@\0001\006��H\025T�\n\n\vK\000\025��^h���\223R>\200\030\000r\213�\000\000\001\001\b\n�$*\200:��\a", '\0' <repeats 75 times>}}
(kgdb) 