help w/panic under heavy load - 5.4

[Edwin]

Hi,

I have a recurring (re-producible) panic on the 5.3/5.4 kernels and I would like to ask
for some help in tracking it down. :) - it could be some misconfig on my part - but
i have tried several different configs of the kernel - ultimately w/ polling on/off,
ipfw on/off, ipfastforwarding on/off - although with ipff off - the box still crashes
but in a different location - it will even crash w/ GENERIC kernel under heavy load.

I'm not quite sure where to look past the below (ie. what variables/etc to present to
the list).

Thanks! /edwin


details are:

- Soekris net4801 platform (SBC w/ Geode processor - i586 class).
- similar crash was seen (as regularly) under 5.3 as well - just never got around to
getting crashdump/etc.
- crash is re-producible under heavy load

here is the crash-info:

fb54c#

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xc
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc0647527
stack pointer           = 0x10:0xc7692b78
frame pointer           = 0x10:0xc7692b9c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 29 (swi1: net)
trap number             = 12
panic: page fault
Uptime: 1m50s
Dumping 128 MB
 16 32 48 64 80 96 112
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort

** and the kgdb output **

mbsd05# uname -a
FreeBSD mbsd05.maplecreek.org 5.4-RELEASE FreeBSD 5.4-RELEASE #0: Fri Jul 15 01:34:05 EDT 2005     root at mbsd05.maplecreek.org:/usr/obj/usr/src/sys/SMP_OPT  i386
mbsd05# kgdb ./kernel.debug /tmp/crash/vmcore.3
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
#0  doadump () at pcpu.h:159
159             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) 
(kgdb) l *0xc0647527
0xc0647527 is in m_copym (/usr/src/sys/kern/uipc_mbuf.c:386).
381             MBUF_CHECKSLEEP(wait);
382             if (off == 0 && m->m_flags & M_PKTHDR)
383                     copyhdr = 1;
384             while (off > 0) {
385                     KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
386                     if (off < m->m_len)
387                             break;
388                     off -= m->m_len;
389                     m = m->m_next;
390             }
(kgdb) 
(kgdb) where
#0  doadump () at pcpu.h:159
#1  0xc0615326 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:410
#2  0xc06155bc in panic (fmt=0xc0819052 "%s") at /usr/src/sys/kern/kern_shutdown.c:566
#3  0xc07d141c in trap_fatal (frame=0xc7692b38, eva=12) at /usr/src/sys/i386/i386/trap.c:817
#4  0xc07d1187 in trap_pfault (frame=0xc7692b38, usermode=0, eva=12) at /usr/src/sys/i386/i386/trap.c:735
#5  0xc07d0dc9 in trap (frame=
      {tf_fs = -1057423336, tf_es = 16, tf_ds = -1057226736, tf_edi = -1056791980, tf_esi = 1432, tf_ebp = -949408868,
 tf_isp = -949408924, tf_ebx = -1056792064, tf_edx = 0, tf_ecx = -1055088626, tf_eax = 0, tf_trapno = 12, tf_err = -1066926080, tf_eip = -1067158233, tf_cs = 8, tf_eflags = 66050, tf_esp = 0, tf_ss = -1065850424})
    at /usr/src/sys/i386/i386/trap.c:425
#6  0xc07c12ca in calltrap () at /usr/src/sys/i386/i386/exception.s:140
#7  0xc0f90018 in ?? ()
#8  0x00000010 in ?? ()
#9  0xc0fc0010 in ?? ()
#10 0xc102a254 in ?? ()
#11 0x00000598 in ?? ()
#12 0xc7692b9c in ?? ()
#13 0xc7692b64 in ?? ()
#14 0xc102a200 in ?? ()
#15 0x00000000 in ?? ()
#16 0xc11ca00e in ?? ()
#17 0x00000000 in ?? ()
#18 0x0000000c in ?? ()
#19 0xc0680000 in pppdealloc (sc=0xc102a200) at /usr/src/sys/net/if_ppp.c:388
#20 0xc06ac168 in ip_fragment (ip=0xc11ca00e, m_frag=0xc7692c48, mtu=-1056791980, if_hwassist_flags=0, sw_csum=1)
    at /usr/src/sys/netinet/ip_output.c:967
#21 0xc06a39c8 in ip_fastforward (m=0xc11b8600) at /usr/src/sys/netinet/ip_fastfwd.c:572
#22 0xc067d65d in ether_demux (ifp=0xc0f91800, m=0xc11b8600) at /usr/src/sys/net/if_ethersubr.c:770
#23 0xc067d419 in ether_input (ifp=0xc0f91800, m=0xc11b8600) at /usr/src/sys/net/if_ethersubr.c:631
#24 0xc0729c2f in sis_rxeof (sc=0xc0f91800) at /usr/src/sys/pci/if_sis.c:1636
#25 0xc0729f6b in sis_poll (ifp=0xc0f91800, cmd=POLL_ONLY, count=0) at /usr/src/sys/pci/if_sis.c:1769
#26 0xc05f8398 in netisr_poll () at /usr/src/sys/kern/kern_poll.c:384
#27 0xc0684f7a in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:338
#28 0xc0601c39 in ithread_loop (arg=0xc0ec6480) at /usr/src/sys/kern/kern_intr.c:547
#29 0xc0600ecc in fork_exit (callout=0xc0601ae8 <ithread_loop>, arg=0xc0ec6480, frame=0xc7692d48)
    at /usr/src/sys/kern/kern_fork.c:791
#30 0xc07c132c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:209
(kgdb)