kern/107865: kernel panic in 6.2-RC2 on heavy network load
Mikle Davidkin
skylord at vt.net.ru
Sat Jan 13 02:00:29 PST 2007
>Number: 107865
>Category: kern
>Synopsis: kernel panic in 6.2-RC2 on heavy network load
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sat Jan 13 10:00:27 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator: Mikle Davidkin
>Release: 6.2-RC2
>Organization:
Vector Telecom
>Environment:
FreeBSD router03.vt.net.ru 6.2-RC2 FreeBSD 6.2-RC2 #0: Mon Jan 8 13:29:30 MSK 2007 root at router03.vt.net.ru:/usr/src/sys/i386/compile/skykernel i386
>Description:
Kernel panics almost every evening when the most heavy network load is on the net. Machine is just a software router for 3 subnets with routed, ipfw, pf (for binat) and no more...
May be it's related to my other PR - 107864, because routers are almost the same and doing similar work just on different subnets of my network...
Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added:
options DEVICE_POLLING
options HZ=1000
options IPFIREWALL
options DUMMYNET
options IPFIREWALL_FORWARD
dmesg
==============
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RC2 #0: Mon Jan 8 13:29:30 MSK 2007
root at router03.vt.net.ru:/usr/src/sys/i386/compile/skykernel
ACPI APIC Table: <Nvidia AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) Processor 2600+ (1607.34-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x1<SSE3>
AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
AMD Features2=0x1<LAHF>
real memory = 268369920 (255 MB)
avail memory = 252952576 (241 MB)
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <NVIDIA nForce3-250 AGP Controller> mem 0xee000000-0xefffffff at device 0.0 on pci0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xf6002000-0xf6002fff irq 20 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xf6003000-0xf6003fff irq 21 at device 2.1 on pci0
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: SMM does not respond, resetting
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem 0xf6004000-0xf60040ff irq 22 at device 2.2 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 8 ports with 8 removable, self powered
nve0: <NVIDIA nForce MCP7 Networking Adapter> port 0xb800-0xb807 mem 0xf6000000-0xf6000fff irq 20 at device 5.0 on pci0
nve0: Ethernet address 00:04:61:a5:08:01
miibus0: <MII bus> on nve0
rlphy0: <RTL8201L 10/100 media interface> on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
nve0: Ethernet address: 00:04:61:a5:08:01
atapci0: <nVidia nForce3 Pro UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9000-0x903f mem 0xf5100000-0xf511ffff,0xf5120000-0xf513ffff irq 19 at device 7.0 on pci2
em0: Ethernet address: 00:07:e9:0f:65:32
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0x9400-0x947f mem 0xf5143000-0xf514307f irq 16 at device 8.0 on pci2
miibus1: <MII bus> on xl0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:0a:5e:4f:f3:d6
xl1: <3Com 3c905C-TX Fast Etherlink XL> port 0x9800-0x987f mem 0xf5140000-0xf514007f irq 17 at device 9.0 on pci2
miibus2: <MII bus> on xl1
ukphy1: <Generic IEEE 802.3u media interface> on miibus2
ukphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl1: Ethernet address: 00:0a:5e:4f:f3:d9
xl2: <3Com 3c905C-TX Fast Etherlink XL> port 0x9c00-0x9c7f mem 0xf5141000-0xf514107f irq 18 at device 10.0 on pci2
miibus3: <MII bus> on xl2
ukphy2: <Generic IEEE 802.3u media interface> on miibus3
ukphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl2: Ethernet address: 00:0a:5e:4f:f3:e2
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xa000-0xa03f mem 0xf5142000-0xf5142fff,0xf5000000-0xf50fffff irq 19 at device 11.0 on pci2
miibus4: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:b7:07:c1:cc
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <Standard parallel printer port> port 0x378-0x37f,0x778-0x77b irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd17ff,0xd2000-0xd27ff,0xd3000-0xd37ff,0xd4000-0xd4fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1607340759 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled
ad0: 38204MB <SAMSUNG SP0411N TW100-13> at ata0-master UDMA100
=================
backtrace
============================
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x78
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc05719ad
stack pointer = 0x28:0xd6e18a7c
frame pointer = 0x28:0xd6e18a80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 306 (routed)
trap number = 12
panic: page fault
Uptime: 2d13h32m45s
Physical memory: 247 MB
Dumping 65 MB: 50 34 18 2
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc05513de in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc0551674 in panic (fmt=0xc07318a5 "%s") at ../../../kern/kern_shutdown.c:565
#3 0xc0708790 in trap_fatal (frame=0xd6e18a3c, eva=120) at ../../../i386/i386/trap.c:837
#4 0xc0707f72 in trap (frame=
{tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 16, tf_esi = -1035997184, tf_ebp = -689862016, tf_isp = -689862040, tf_ebx = -1032266368, tf_edx = -1032266368, tf_ecx = 4, tf_eax = -1035997152, tf_trapno = 12, tf_err = 0, tf_eip = -1068033619, tf_cs = 32, tf_eflags = 65539, tf_esp = -1035997184, tf_ss = -689861980})
at ../../../i386/i386/trap.c:270
#5 0xc06f712a in calltrap () at ../../../i386/i386/exception.s:139
#6 0xc05719ad in turnstile_setowner (ts=0xc278dd80, owner=0x4) at ../../../kern/subr_turnstile.c:432
#7 0xc0571ca4 in turnstile_wait (lock=0xc2809168, owner=0x4) at ../../../kern/subr_turnstile.c:591
#8 0xc0547b60 in _mtx_lock_sleep (m=0xc2809168, tid=3258970112, opts=0, file=0x0, line=0)
at ../../../kern/kern_mutex.c:579
#9 0xc05cdcdc in rt_setgate (rt=0xc27e0bdc, dst=0xc27b40e0, gate=0xc363b66c) at ../../../net/route.c:1041
#10 0xc05ceecd in route_output (m=0xc36fed00, so=0xc247b000) at ../../../net/rtsock.c:487
#11 0xc05cc634 in raw_usend (so=0x4, flags=0, m=0xc278dd80, nam=0x0, control=0xc23ff020, td=0xc23ff000)
at ../../../net/raw_usrreq.c:263
#12 0xc05ce7cb in rts_send (so=0xc247b000, flags=0, m=0xc36fed00, nam=0x0, control=0x0, td=0xc23ff000)
at ../../../net/rtsock.c:269
#13 0xc058a4d3 in sosend (so=0xc247b000, addr=0x0, uio=0xd6e18cbc, top=0xc36fed00, control=0x0, flags=0,
td=0xc23ff000) at ../../../kern/uipc_socket.c:836
#14 0xc0578d5e in soo_write (fp=0xc23ff020, uio=0xd6e18cbc, active_cred=0xc2173d80, flags=0,
td=0xc23ff000) at ../../../kern/sys_socket.c:118
#15 0xc0573467 in dofilewrite (td=0xc23ff000, fd=3, fp=0xc2424288, auio=0xd6e18cbc, offset=Unhandled dwarf expression opcode 0x93
) at file.h:252
#16 0xc057330b in kern_writev (td=0xc23ff000, fd=3, auio=0xd6e18cbc) at ../../../kern/sys_generic.c:402
#17 0xc0573231 in write (td=0xc23ff000, uap=0x4) at ../../../kern/sys_generic.c:326
#18 0xc0708aa7 in syscall (frame=
{tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1024, tf_esi = 3, tf_ebp = -1077941176, tf_isp = -689861276, tf_ebx = 1, tf_edx = -1077941220, tf_ecx = 2, tf_eax = 4, tf_trapno = 22, tf_err = 2, tf_eip = 672494419, tf_cs = 51, tf_eflags = 642, tf_esp = -1077941380, tf_ss = 59}) at ../../../i386/i386/trap.c:983
#19 0xc06f717f in Xint0x80_syscall () at ../../../i386/i386/exception.s:200
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) list *0xc05719ad
0xc05719ad is in turnstile_setowner (../../../kern/subr_turnstile.c:433).
428 * Malloc a turnstile for a new thread, initialize it and return it.
429 */
430 struct turnstile *
431 turnstile_alloc(void)
432 {
433 struct turnstile *ts;
434
435 ts = malloc(sizeof(struct turnstile), M_TURNSTILE, M_WAITOK | M_ZERO);
436 TAILQ_INIT(&ts->ts_blocked);
437 TAILQ_INIT(&ts->ts_pending);
>How-To-Repeat:
panic occurs in 1-3 days uptume
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list