Kernel Panic
Erich Jenkins
erich at fuujingroup.com
Thu Apr 9 16:31:44 PDT 2009
I'm hoping someone can shed some light on this issue. I've posted to
several lists with no responses. We're running a 7.0 server with a
customer kernel (see below) but the results have been the same on
GENERIC, and on i386 hardware vs amd64 hardware (both hardware and
kernels). This server is responsible for NAT for a few subnets behind
it, and it running multiple instances of OpenVPN for connectivity to the
subnets (bridged, not routed VPN). The box panics and reboots frequently
(perhaps 2 times daily). I found a bug report in GNATS that looks
similar to this panic (See GNATS 129316 or 127699 for example). All of
these faults seem to be similar and the bt for each looks VERY similar.
What am I missing here?? Any assistance would be greatly appreciated at
this point.
Here's the kernel info for this box:
cpu I686_CPU
ident KNL04012009
makeoptions DEBUG=-g # Build kernel with gdb(1) debug
symbols
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options SCTP # Stream Control Transmission
Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options UFS_GJOURNAL # Enable gjournal-based UFS
journaling
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires
NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUS using NMI instead of IPI
options AUDIT # Security event auditing
options IPFIREWALL
options IPDIVERT
# To make an SMP kernel, the next two lines are needed
options SMP # Symmetric MultiProcessor Kernel
device apic # I/O APIC
# Bus support.
device eisa
device pci
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
device agp # support several AGP chipsets
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
device uart # Generic UART driver
# PCI Ethernet NICs.
device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 adapter Gigabit
Ethernet Card
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device pcn # AMD Am79C97x PCI 10/100 (precedence
over 'le')
device rl # RealTek 8129/8139
device sis # Silicon Integrated Systems SiS 900/SiS
7016
device vr # VIA Rhine, Rhine II
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
device bpf # Berkeley packet filter
Here's the dmesg output (after a panic and reboot):
FreeBSD 7.0-RELEASE #0: Wed Apr 1 22:01:11 CDT 2009
ibe7boo at vpngw.fuujinnetworks.com:/usr/src/sys/i386/compile/KNL04012009
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) 64 Processor 3000+ (1999.80-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x1<SSE3>
AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
AMD Features2=0x1<LAHF>
real memory = 787349504 (750 MB)
avail memory = 760778752 (725 MB)
ACPI APIC Table: <VIAK8M AWRDACPI>
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <VIAK8M AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 2ede0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
powernow0: <Cool`n'Quiet K8> on cpu0
device_attach: powernow0 attach returned 6
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8380 host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem
0xf4000000-0xf7ffffff,0xfa000000-0xfaffffff irq 16 at device 0.0 on pci1
pcib2: <PCI-PCI bridge> at device 8.0 on pci0
pci2: <PCI bus> on pcib2
de0: <Digital 21140A Fast Ethernet> port 0xc000-0xc07f mem
0xfc043000-0xfc04307f irq 16 at device 4.0 on pci2
de0: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2
de0: using obsoleted if_watchdog interface
de0: Ethernet address: 00:00:d1:1f:cb:d5
de0: [ITHREAD]
de1: <Digital 21140A Fast Ethernet> port 0xc100-0xc17f mem
0xfc040000-0xfc04007f irq 17 at device 5.0 on pci2
de1: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2
de1: using obsoleted if_watchdog interface
de1: Ethernet address: 00:00:d1:1f:cb:d6
de2: <Digital 21140A Fast Ethernet> port 0xc200-0xc27f mem
0xfc041000-0xfc04107f irq 18 at device 6.0 on pci2
de2: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2
de2: using obsoleted if_watchdog interface
de2: Ethernet address: 00:00:d1:1f:cb:d7
de3: <Digital 21140A Fast Ethernet> port 0xc300-0xc37f mem
0xfc042000-0xfc04207f irq 19 at device 7.0 on pci2
de3: Cogent EM440TX 21140A [10-100Mb/s] pass 2.2
de3: using obsoleted if_watchdog interface
de3: Ethernet address: 00:00:d1:1f:cb:d8
atapci0: <VIA 6420 SATA150 controller> port
0xdb00-0xdb07,0xe100-0xe103,0xd800-0xd807,0xd900-0xd903,0xda00-0xda0f,0xd000-0xd0ff
irq 20 at device 15.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xdc00-0xdc0f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xd400-0xd4ff mem
0xfc101000-0xfc1010ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
miibus0: <MII bus> on vr0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: using obsoleted if_watchdog interface
vr0: Ethernet address: 00:19:21:46:2d:56
vr0: [ITHREAD]
acpi_tz0: <Thermal Zone> on acpi0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xcc000-0xcdfff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
Timecounter "TSC" frequency 1999795935 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, rule-based forwarding
disabled, default to deny, logging disabled
acd0: CDRW <LITE-ON LTR-24102B/5QSB> at ata0-master UDMA33
ad4: 152627MB <WDC WD1600AAJS-00PSA0 05.06H05> at ata2-master SATA150
Trying to mount root from ufs:/dev/ad4s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled
Here's the debug bt:
kgdb /usr/src/sys/i386/compile/KNL04072009/kernel.debug /var/crash/vmcore.1
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x4
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc388994b
stack pointer = 0x28:0xdceb59c8
frame pointer = 0x28:0xdceb5a44
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 26 (irq23: vr0)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 5d7h9m18s
Physical memory: 742 MB
Dumping 119 MB: 104 88 72 56 40 24 8
#0 doadump () at pcpu.h:195
195 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc05ba657 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc05ba919 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:563
#3 0xc080c54c in trap_fatal (frame=0xdceb5988, eva=4) at
../../../i386/i386/trap.c:899
#4 0xc080c7b0 in trap_pfault (frame=0xdceb5988, usermode=0, eva=4) at
../../../i386/i386/trap.c:812
#5 0xc080d132 in trap (frame=0xdceb5988) at ../../../i386/i386/trap.c:490
#6 0xc07f3abb in calltrap () at ../../../i386/i386/exception.s:139
#7 0xc388994b in ?? ()
Previous frame inner to this frame (corrupt stack?)
Here's a pciconf -lv output:
hostb0 at pci0:0:0:0: class=0x060000 card=0x02041106 chip=0x02041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'K8M400 CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
hostb1 at pci0:0:0:1: class=0x060000 card=0x00000000 chip=0x12041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'K8M400 CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
hostb2 at pci0:0:0:2: class=0x060000 card=0x00000000 chip=0x22041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'K8M400 CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
hostb3 at pci0:0:0:3: class=0x060000 card=0x00000000 chip=0x32041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = '1394 i2c CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
hostb4 at pci0:0:0:4: class=0x060000 card=0x00000000 chip=0x42041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'K8M400 CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
hostb5 at pci0:0:0:7: class=0x060000 card=0x00000000 chip=0x72041106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'K8M400 CPU to PCI Bridge'
class = bridge
subclass = HOST-PCI
pcib1 at pci0:0:1:0: class=0x060400 card=0x00000000 chip=0xb1881106
rev=0x00 hdr=0x01
vendor = 'VIA Technologies Inc'
device = 'VT8237 K8HTB CPU to AGP 2.0/3.0 Bridge'
class = bridge
subclass = PCI-PCI
pcib2 at pci0:0:8:0: class=0x060400 card=0x00000000 chip=0x00241011
rev=0x03 hdr=0x01
vendor = 'Digital Equipment Corporation'
device = '21151/2 PCI to PCI Bridge'
class = bridge
subclass = PCI-PCI
atapci0 at pci0:0:15:0: class=0x01018f card=0x32061565 chip=0x31491106
rev=0x80 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'VT8237 VT6410 SATA RAID Controller'
class = mass storage
subclass = ATA
atapci1 at pci0:0:15:1: class=0x01018a card=0x32061565 chip=0x05711106
rev=0x06 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'VT82C586A/B/VT82C686/A/B/VT823x/A/C Bus Master IDE
Controller'
class = mass storage
subclass = ATA
isab0 at pci0:0:17:0: class=0x060100 card=0x32061565 chip=0x32271106
rev=0x00 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'VT8237 PCI-to-ISA Bridge'
class = bridge
subclass = PCI-ISA
vr0 at pci0:0:18:0: class=0x020000 card=0x01021106 chip=0x30651106
rev=0x78 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'VT6102 Rhine II PCI Fast Ethernet Controller||Used by
GERICOM in laptop Webengine Advanced'
class = network
subclass = ethernet
hostb6 at pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022
rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron HyperTransport Technology
Configuration'
class = bridge
subclass = HOST-PCI
hostb7 at pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022
rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron Address Map'
class = bridge
subclass = HOST-PCI
hostb8 at pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022
rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron DRAM Controller'
class = bridge
subclass = HOST-PCI
hostb9 at pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022
rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices (AMD)'
device = '(K8) Athlon 64/Opteron Miscellaneous Control'
class = bridge
subclass = HOST-PCI
vgapci0 at pci0:1:0:0: class=0x030000 card=0x12031565 chip=0x31081106
rev=0x01 hdr=0x00
vendor = 'VIA Technologies Inc'
device = 'Unknown Via Unichrome S3 VGA - part of a VIA Northbridge'
class = display
subclass = VGA
de0 at pci0:2:4:0: class=0x020000 card=0x24001109 chip=0x00091011 rev=0x22
hdr=0x00
vendor = 'Digital Equipment Corporation'
device = 'DecChip 21140 Fast Ethernet Adapter'
class = network
subclass = ethernet
de1 at pci0:2:5:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22
hdr=0x00
vendor = 'Digital Equipment Corporation'
device = 'DecChip 21140 Fast Ethernet Adapter'
class = network
subclass = ethernet
de2 at pci0:2:6:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22
hdr=0x00
vendor = 'Digital Equipment Corporation'
device = 'DecChip 21140 Fast Ethernet Adapter'
class = network
subclass = ethernet
de3 at pci0:2:7:0: class=0x020000 card=0x00000000 chip=0x00091011 rev=0x22
hdr=0x00
vendor = 'Digital Equipment Corporation'
device = 'DecChip 21140 Fast Ethernet Adapter'
class = network
subclass = ethernet
Thanks in advance folks! I appreciate any assistance offered.
--
Erich M. Jenkins
Fuujin Group Limited
PO Box 792
Brainerd, MN 56401
(p) 218-824-5038
(f) 218-824-7516
Visit us on the web at http://www.fuujingroup.com
"You should never, never doubt what no one is sure about."
-- Gene Wilder
More information about the freebsd-bugs
mailing list