kern/103619: Kernel panic (page fault) during normal operation

[Daniele Pilenga]

>Number:         103619
>Category:       kern
>Synopsis:       Kernel panic (page fault) during normal operation
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Sep 25 13:30:30 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Daniele Pilenga
>Release:        FreeBSD 6.1-RELEASE-p7
>Organization:
Atos Origin Italia s.p.a.
>Environment:
FreeBSD srvbsd01.gruppo.bipielle 6.1-RELEASE-p7 FreeBSD 6.1-RELEASE-p7 #0: Wed Sep 20 18:02:58 CEST 2006     root at srvbsd01.gruppo.bipielle:/usr/obj/usr/src/sys/SRVBSD  i386
>Description:
During normal operation, no particular stress or other activities involved, the kernel panics.

It's an HP DL360G3, with 2 Xeon 3.2GHz and 2GB of ram acting as squid, bind, dhcp server, 4 carp interfaces defined as fail over with an other identical machine which exhibits the same problem.


Kernel config follows:

include GENERIC
makeoptions     DEBUG=-g
options         SMP
ident           SRVBSD
nocpu           I486_CPU
nocpu           I586_CPU
device          carp
device          pf
device          pflog


kgdb follows:

kgdb /boot/kernel/kernel.debug  vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 06
fault virtual address   = 0x4
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0719e1b
stack pointer           = 0x28:0xe8e608d8
frame pointer           = 0x28:0xe8e608e8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 64389 (sockstat)
trap number             = 12
panic: page fault
cpuid = 1
Uptime: 1d18h59m4s
Dumping 2047 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 2047MB (524026 pages) 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 18
24 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 15
04 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 11
84 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 83
2 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 43
2 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16

#0  doadump () at pcpu.h:165
165     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) where
#0  doadump () at pcpu.h:165
#1  0xc06d10ed in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2  0xc06d149e in panic (fmt=0xc0969b52 "%s") at /usr/src/sys/kern/kern_shutdown.c:558
#3  0xc091a655 in trap_fatal (frame=0xe8e60898, eva=0) at /usr/src/sys/i386/i386/trap.c:836
#4  0xc091a349 in trap_pfault (frame=0xe8e60898, usermode=0, eva=4)
    at /usr/src/sys/i386/i386/trap.c:744
#5  0xc0919f2f in trap (frame=
      {tf_fs = -1064304632, tf_es = -969080792, tf_ds = -387579864, tf_edi = 1542, tf_esi = 0, tf_eb
p = -387577624, tf_isp = -387577660, tf_ebx = -387577072, tf_edx = -387577396, tf_ecx = 0, tf_eax = 
0, tf_trapno = 12, tf_err = 0, tf_eip = -1066295781, tf_cs = 32, tf_eflags = 66198, tf_esp = -387577
072, tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:434
#6  0xc090546a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc0719e1b in sotoxsocket (so=0x0, xso=0xe8e60b10) at /usr/src/sys/kern/uipc_socket2.c:1454
#8  0xc0799eb4 in tcp_pcblist (oidp=0xc0a02f40, arg1=0x0, arg2=0, req=0xe8e60bf8)
    at /usr/src/sys/netinet/tcp_subr.c:964
#9  0xc06daba2 in sysctl_root (oidp=0x0, arg1=0x0, arg2=0, req=0xe8e60bf8)
    at /usr/src/sys/kern/kern_sysctl.c:1285
#10 0xc06dae3c in userland_sysctl (td=0x0, name=0xe8e60c68, namelen=4, old=0xe8e60bf8, 
    oldlenp=0xbfbfedf8, inkernel=0, new=0x0, newlen=0, retval=0xe8e60c64, flags=0)
    at /usr/src/sys/kern/kern_sysctl.c:1384
#11 0xc06dac8d in __sysctl (td=0x0, uap=0xe8e60d04) at /usr/src/sys/kern/kern_sysctl.c:1319
#12 0xc091aa3c in syscall (frame=
      {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1077940744, tf_esi = -1077940928, tf_ebp = -107
7940984, tf_isp = -387576476, tf_ebx = 672416808, tf_edx = 0, tf_ecx = 134594560, tf_eax = 202, tf_t
rapno = 12, tf_err = 2, tf_eip = 672265851, tf_cs = 51, tf_eflags = 663, tf_esp = -1077941028, tf_ss
 = 59}) at /usr/src/sys/i386/i386/trap.c:981
#13 0xc09054bf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#14 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 7
#7  0xc0719e1b in sotoxsocket (so=0x0, xso=0xe8e60b10) at /usr/src/sys/kern/uipc_socket2.c:1454
1454            xso->xso_so = so;
(kgdb) p so
$1 = (struct socket *) 0x0

It seems the problem is here, but my understanding of the kernel are too limited.

I can do further tests if needed.
>How-To-Repeat:
The problem started after my first uses of the carp interface, but I have only  recently upgraded from 5.4 and I can not be sure this is the only difference.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: