kern/102412: kernel panic in 6.1-stable during normal operation
Daniel Austin
daniel at kewlio.net
Wed Aug 23 01:00:36 UTC 2006
>Number: 102412
>Category: kern
>Synopsis: kernel panic in 6.1-stable during normal operation
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Aug 23 01:00:31 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Daniel Austin
>Release: 6.1-STABLE (CVS-20060822)
>Organization:
Kewlio.net Limited
>Environment:
FreeBSD <hostname> 6.1-STABLE FreeBSD 6.1-STABLE #6: Tue Aug 22 21:09:55 BST 2006 dan@<hostname>:/usr/obj/usr/src/sys/kewlio i386
>Description:
The server is a dedicated IRC server on one of the big 4 networks.
After a few hours of normal operation, the kernel panics.
I compiled the kernel with WITNESS and INVARIANTS enabled after the original kernel page faults.
Crash debug (with WITNESS/INVARIANTS enabled):
Unread portion of the kernel message buffer:
panic: mtx_lock() of destroyed mutex @ /usr/src/sys/netinet/ip_output.c:1193
Uptime: 2h51m41s
Dumping 503 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc0655934 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc0655bb2 in panic (fmt=0xc087b45c "mtx_lock() of destroyed mutex @ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193)
at /usr/src/sys/kern/kern_mutex.c:281
#4 0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
#5 0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
#6 0xc068f6f4 in sosetopt (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563
#7 0xc0694045 in kern_setsockopt (td=0xc4bfd480, s=1244, level=0, name=0, val=0x0, valseg=UIO_USERSPACE, valsize=0)
at /usr/src/sys/kern/uipc_syscalls.c:1351
#8 0xc0693f8e in setsockopt (td=0xc4bfd480, uap=0x0) at /usr/src/sys/kern/uipc_syscalls.c:1307
#9 0xc080fdcf in syscall (frame=
{tf_fs = 59, tf_es = -1078001605, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 1244, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 1244, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59})
at /usr/src/sys/i386/i386/trap.c:981
#10 0xc07fed2f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#11 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 3
#3 0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193)
at /usr/src/sys/kern/kern_mutex.c:281
281 KASSERT(m->mtx_lock != MTX_DESTROYED,
(kgdb) f 4
#4 0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
1193 INP_LOCK(inp);
(kgdb) f 5
#5 0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
1038 error = ip_ctloutput(so, sopt);
Crash debug (without WITNESS/INVARIANTS):
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x78
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0681f29
stack pointer = 0x28:0xe63e8ab8
frame pointer = 0x28:0xe63e8abc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 624 (ircd.200608220145)
trap number = 12
panic: page fault
Uptime: 14h17m13s
Dumping 503 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc0661ad6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc0661d6c in panic (fmt=0xc08ac642 "%s") at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc0861034 in trap_fatal (frame=0xe63e8a78, eva=120) at /usr/src/sys/i386/i386/trap.c:836
#4 0xc0860816 in trap (frame=
{tf_fs = -432144376, tf_es = -994115544, tf_ds = -994115544, tf_edi = 0, tf_esi = -994102144, tf_ebp = -432108868, tf_isp = -432108892, tf_ebx = -994097216, tf_edx = -994097216, tf_ecx = 4, tf_eax = -994102112, tf_trapno = 12, tf_err = 0, tf_eip = -1066918103, tf_cs = 32, tf_eflags = 65543, tf_esp = -994102144, tf_ss = -432108832}) at /usr/src/sys/i386/i386/trap.c:269
#5 0xc084fa8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6 0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432
#7 0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591
#8 0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#9 0xc07090ec in ip_ctloutput (so=0xc4bf34a0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193
#10 0xc071971f in tcp_ctloutput (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038
#11 0xc069bb10 in sosetopt (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563
#12 0xc06a0dfd in kern_setsockopt (td=0xc4bf3480, s=2662, level=-994102112, name=-994102112, val=0xc4bf47c0, valseg=UIO_USERSPACE, valsize=4)
at /usr/src/sys/kern/uipc_syscalls.c:1351
#13 0xc06a0d2e in setsockopt (td=0xc4bf3480, uap=0xc4bf34a0) at /usr/src/sys/kern/uipc_syscalls.c:1307
#14 0xc086134b in syscall (frame=
{tf_fs = 198377531, tf_es = 223215675, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 2662, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 2662, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59})
at /usr/src/sys/i386/i386/trap.c:981
#15 0xc084fadf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 6
#6 0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432
432 ts->ts_owner = owner;
(kgdb) f 7
#7 0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591
591 turnstile_setowner(ts, owner);
(kgdb) f 8
#8 0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
579 turnstile_wait(&m->mtx_object, mtx_owner(m));
(kgdb) print *m
$1 = {mtx_object = {lo_class = 0xc093be44, lo_name = 0xc08d60d8 "inp", lo_type = 0xc08d3871 "tcpinp", lo_flags = 4849664, lo_list = {tqe_next = 0x0,
tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 6, mtx_recurse = 0}
>How-To-Repeat:
I have not been able to duplicate this problem on another function machine, however I installed and moved the IRC server to different hardware and the same problem occurs. This happens approximately every 3 hours and seems related to the level of network traffic.
The server has fxp and em-based cards with polling enabled.
I am happy to work with anyone that needs any further information or assistance in diagnosing the problem. The problem started since updating from CVS recently (a CVS build from 20060623 did not cause a kernel panic, but did cause a deadlock in the ircd process)
Copies of vmcores/kernels with debugging symbols have been kept.
>Fix:
Unsure.
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list