kern/104765: kernel panic 6.2 prerelease-20061017 amd64

Tue Oct 24 11:40:26 PDT 2006

>Number:         104765
>Category:       kern
>Synopsis:       kernel panic 6.2 prerelease-20061017 amd64
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 24 18:40:17 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Kai Gallasch
>Release:        6.2 prerelease (checkout 20061017)
>Organization:
FREE!
>Environment:
FreeBSD geldkraft.free.de 6.2-PRERELEASE FreeBSD 6.2-PRERELEASE #0: Sun Oct 22 13:36:38 CEST 2006     houdini at geldkraft.free.de:/usr/obj/usr/src/sys/SMP  amd64
>Description:
Kernel panics after 1-3 days uptime with trap number 12 - page fault.

kernel config:
--------------
GENERIC (SMP) with "makeoptions DEBUG=-g"
$FreeBSD: src/sys/amd64/conf/GENERIC,v 1.439.2.14 2006/10/09 18:41:36 simon Exp $

Hardware:
---------

HP/Compaq DL385 Dual Opteron (Dual Core) with ServeRaid 6 (Raid 5) and 1G RAM.

dmesg:
------

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-PRERELEASE #0: Sun Oct 22 13:36:38 CEST 2006
    houdini at geldkraft.free.de:/usr/obj/usr/src/sys/SMP
ACPI APIC Table: <HP     00000083>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 280 (2405.47-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f12  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x2<CMP>
  Cores per package: 2
real memory  = 1073709056 (1023 MB)
avail memory = 1023938560 (976 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
ioapic3 <Version 1.1> irqs 32-35 on motherboard
ioapic4 <Version 1.1> irqs 36-39 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <HP A05> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ohci0: <OHCI (generic) USB controller> mem 0xf7df0000-0xf7df0fff irq 19 at device 0.0 on pci1
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xf7de0000-0xf7de0fff irq 19 at device 0.1 on pci1
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: SMM does not respond, resetting
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
pci1: <base peripheral> at device 2.0 (no driver attached)
pci1: <base peripheral> at device 2.2 (no driver attached)
pci1: <display, VGA> at device 3.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <AMD 8111 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 4.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <bridge> at device 4.3 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ciss0: <HP Smart Array 6i> port 0x5000-0x50ff mem 0xf7ef0000-0xf7ef1fff,0xf7e80000-0xf7ebffff irq 24 at device 4.0 on pci2
ciss0: [GIANT-LOCKED]
pci0: <base peripheral, interrupt controller> at device 7.1 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7ff0000-0xf7ffffff irq 28 at device 6.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge0: Ethernet address: 00:17:a4:8f:27:68
bge1: <Broadcom BCM5704 B0, ASIC rev. 0x2100> mem 0xf7fe0000-0xf7feffff irq 29 at device 6.1 on pci3
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge1: Ethernet address: 00:17:a4:8f:27:67
pci0: <base peripheral, interrupt controller> at device 8.1 (no driver attached)
pcib4: <ACPI Host-PCI bridge> on acpi0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 9.0 on pci4
pci5: <ACPI PCI bus> on pcib5
pci4: <base peripheral, interrupt controller> at device 9.1 (no driver attached)
pcib6: <ACPI PCI-PCI bridge> at device 10.0 on pci4
pci6: <ACPI PCI bus> on pcib6
pci4: <base peripheral, interrupt controller> at device 10.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: [FAST]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xee000-0xeffff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDROM <HL-DT-ST GCR-8240N/2.03> at ata0-master PIO4
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da0: 135.168MB/s transfers
da0: 17200MB (35226720 512 byte sectors: 255H 32S/T 4317C)
da1 at ciss0 bus 0 target 1 lun 0
da1: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da1: 135.168MB/s transfers
da1: 17200MB (35226720 512 byte sectors: 255H 32S/T 4317C)
da2 at ciss0 bus 0 target 2 lun 0
da2: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da2: 135.168MB/s transfers
da2: 69499MB (142334880 512 byte sectors: 255H 32S/T 17443C)
da3 at ciss0 bus 0 target 3 lun 0
da3: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da3: 135.168MB/s transfers
da3: 69499MB (142334880 512 byte sectors: 255H 32S/T 17443C)
da4 at ciss0 bus 0 target 4 lun 0
da4: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da4: 135.168MB/s transfers
da4: 139799MB (286309920 512 byte sectors: 255H 32S/T 35087C)

backtrace:
----------

[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:
d, page not present
instruction pointer     = 0x8:0xffffffff803eea47
stack pointer           = 0x10:0xffffffffa814a8b0
frame pointer           = 0x10:0x4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 27596 (tcpserver)
trap number             = 12
panic: page fault
cpuid = 3
Uptime: 2h12m0s
Dumping 1023 MB (2 chunks)
  chunk 0: 1MB (156 pages) ... ok
  chunk 1: 1023MB (261880 pages) 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16

#0  doadump () at pcpu.h:172
172     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) quit
geldkraft:/etc # mount /usr/src/
geldkraft:/etc # cd /usr/src/sys/amd64/conf/
geldkraft:/usr/src/sys/amd64/conf # kgdb SMP  /var/crash/vmcore.0
kgdb: bad namelist - no kernbase
geldkraft:/usr/src/sys/amd64/conf # kgdb /boot/kernel/kernel /var/crash/vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:
d, page not present
instruction pointer     = 0x8:0xffffffff803eea47
stack pointer           = 0x10:0xffffffffa814a8b0
frame pointer           = 0x10:0x4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 27596 (tcpserver)
trap number             = 12
panic: page fault
cpuid = 3
Uptime: 2h12m0s
Dumping 1023 MB (2 chunks)
  chunk 0: 1MB (156 pages) ... ok
  chunk 1: 1023MB (261880 pages) 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16

#0  doadump () at pcpu.h:172
172     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) list *0xffffffff803eea47
0xffffffff803eea47 is in _mtx_lock_sleep (/usr/src/sys/kern/kern_mutex.c:548).
543                      * If the current owner of the lock is executing on another
544                      * CPU, spin instead of blocking.
545                      */
546                     owner = (struct thread *)(v & MTX_FLAGMASK);
547     #ifdef ADAPTIVE_GIANT
548                     if (TD_IS_RUNNING(owner)) {
549     #else
550                     if (m != &Giant && TD_IS_RUNNING(owner)) {
551     #endif
552                             turnstile_release(&m->mtx_object);
(kgdb) backtrace
#0  doadump () at pcpu.h:172
#1  0x0000000000000004 in ?? ()
#2  0xffffffff803f8fd7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#3  0xffffffff803f9671 in panic (fmt=0xffffff0002116980 "X?J:") at /usr/src/sys/kern/kern_shutdown.c:565
#4  0xffffffff80618b3f in trap_fatal (frame=0xffffff0002116980, eva=18446742975175902040) at /usr/src/sys/amd64/amd64/trap.c:660
#5  0xffffffff80619066 in trap (frame=
      {tf_rdi = 11, tf_rsi = -1099476932224, tf_rdx = 6, tf_rcx = 0, tf_r8 = 4, tf_r9 = -1098475933086, tf_rax = 1, tf_rbx = -1099415090280, tf_rbp = 4, tf_r10 = 4, tf_r11 = 4, tf_r12 = -1099476932224, tf_r13 = -1098728017152, tf_r14 = 0, tf_r15 = 1, tf_trapno = 12, tf_addr = 396, tf_flags = -2141616351, tf_err = 0, tf_rip = -2143360441, tf_cs = 8, tf_rflags = 65538, tf_rsp = -1475041088, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:238
#6  0xffffffff8060442b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:168
#7  0xffffffff803eea47 in _mtx_lock_sleep (m=0xffffff0005c10b98, tid=18446742974232619392, opts=6, file=0x0, line=4)
    at /usr/src/sys/kern/kern_mutex.c:546
#8  0xffffffff804bb51d in ip_ctloutput (so=0xb, sopt=0xffffffffa814ab30) at /usr/src/sys/netinet/ip_output.c:1193
#9  0xffffffff804ccad5 in tcp_ctloutput (so=0xffffff0024a0d268, sopt=0xffffffffa814ab30) at /usr/src/sys/netinet/tcp_usrreq.c:1038
#10 0xffffffff804416b8 in sosetopt (so=0xffffff0024a0d268, sopt=0xffffffffa814ab30) at /usr/src/sys/kern/uipc_socket.c:1563
#11 0xffffffff80447b93 in kern_setsockopt (td=0xffffff0002116980, s=616888072, level=4, name=0, val=0x4, valseg=1035694690, valsize=11)
    at /usr/src/sys/kern/uipc_syscalls.c:1351
#12 0xffffffff80447bfe in setsockopt (td=0xb, uap=0xffffff0002116980) at /usr/src/sys/kern/uipc_syscalls.c:1307
#13 0xffffffff80619991 in syscall (frame=
      {tf_rdi = 0, tf_rsi = 0, tf_rdx = 1, tf_rcx = 0, tf_r8 = 0, tf_r9 = 140737488350072, tf_rax = 105, tf_rbx = 0, tf_rbp = 3, tf_r10 = -3689348814741910323, tf_r11 = 514, tf_r12 = 140737488350480, tf_r13 = 34368406752, tf_r14 = 0, tf_r15 = 0, tf_trapno = 12, tf_addr = 5283944, tf_flags = 12, tf_err = 2, tf_rip = 34366834188, tf_cs = 43, tf_rflags = 518, tf_rsp = 140737488350184, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:792
#14 0xffffffff806045c8 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:270
#15 0x00000008006c460c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

>How-To-Repeat:
problem occurs in between 1-3 days uptime of server
>Fix:
Raising some sysctl values seems to lengthen the intervals between crashes.
Although I might be mistaken that tweaking them has some effect on the problem.

# default war 12328
#kern.maxfiles=80000

# default 128
#kern.ipc.somaxconn=384

# default war 11095
#kern.maxfilesperproc=50000

>Release-Note:
>Audit-Trail:
>Unformatted: