panic in deadlkres() on r267110
Glen Barber
gjb at FreeBSD.org
Fri Jun 6 14:12:20 UTC 2014
Two machines in the cluster panic last night with the same backtrace.
It is unclear yet exactly what was happening on the systems, but both
are port building machines using ports-mgmt/tinderbox.
Any ideas or information on how to further debug this would be
appreciated.
Script started on Fri Jun 6 14:01:53 2014
command: /bin/sh
# uname -a
FreeBSD redbuild04.nyi.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r267110: Thu Jun 5 15:57:43 UTC 2014 sbruno at redbuild04.nyi.freebsd.org:/usr/obj/usr/src/sys/REDBUILD amd64
# kgdb ./kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: deadlkres: possible deadlock detected on allproc_lock
cpuid = 17
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1838702a20
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1838702ad0
panic() at panic+0x155/frame 0xfffffe1838702b50
deadlkres() at deadlkres+0x42a/frame 0xfffffe1838702bb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe1838702bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1838702bf0
--- trap 0, rip = 0, rsp = 0xfffffe1838702cb0, rbp = 0 ---
KDB: enter: panic
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
#0 doadump (textdump=-946873840) at pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0 doadump (textdump=-946873840) at pcpu.h:219
#1 0xffffffff8034e865 in db_fncall (dummy1=<value optimized out>,
dummy2=<value optimized out>, dummy3=<value optimized out>,
dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:578
#2 0xffffffff8034e54d in db_command (cmd_table=0x0)
at /usr/src/sys/ddb/db_command.c:449
#3 0xffffffff8034e2c4 in db_command_loop ()
at /usr/src/sys/ddb/db_command.c:502
#4 0xffffffff80350d20 in db_trap (type=<value optimized out>, code=0)
at /usr/src/sys/ddb/db_main.c:231
#5 0xffffffff809a9bd9 in kdb_trap (type=3, code=0, tf=<value optimized out>)
at /usr/src/sys/kern/subr_kdb.c:656
#6 0xffffffff80dc00e3 in trap (frame=0xfffffe1838702a00)
at /usr/src/sys/amd64/amd64/trap.c:551
#7 0xffffffff80da29c2 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff809a933e in kdb_enter (why=0xffffffff81039a72 "panic",
msg=<value optimized out>) at cpufunc.h:63
#9 0xffffffff8096a8b5 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:749
#10 0xffffffff8090d16a in deadlkres () at /usr/src/sys/kern/kern_clock.c:203
#11 0xffffffff8093170a in fork_exit (callout=0xffffffff8090cd40 <deadlkres>,
arg=0x0, frame=0xfffffe1838702c00) at /usr/src/sys/kern/kern_fork.c:977
---Type <return> to continue, or q <return> to quit---
#12 0xffffffff80da2efe in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:605
#13 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb) fr 10
#10 0xffffffff8090d16a in deadlkres () at /usr/src/sys/kern/kern_clock.c:203
203 panic("%s: possible deadlock detected on allproc_lock\n",
(kgdb) l
198 * priority inversion problem leading to starvation.
199 * If the lock can't be held after 100 tries, panic.
200 */
201 if (!sx_try_slock(&allproc_lock)) {
202 if (tryl > 100)
203 panic("%s: possible deadlock detected on allproc_lock\n",
204 __func__);
205 tryl++;
206 pause("allproc", sleepfreq * hz);
207 continue;
(kgdb) up
#11 0xffffffff8093170a in fork_exit (callout=0xffffffff8090cd40 <deadlkres>,
arg=0x0, frame=0xfffffe1838702c00) at /usr/src/sys/kern/kern_fork.c:977
977 callout(arg, frame);
(kgdb) l
972 * cpu_set_fork_handler intercepts this function call to
973 * have this call a non-return function to stay in kernel mode.
974 * initproc has its own fork handler, but it does return.
975 */
976 KASSERT(callout != NULL, ("NULL callout in fork_exit"));
977 callout(arg, frame);
978
979 /*
980 * Check if a kernel thread misbehaved and returned from its main
981 * function.
(kgdb) list *0xffffffff8090cd40
0xffffffff8090cd40 is in deadlkres (/usr/src/sys/kern/kern_clock.c:185).
180 static int blktime_threshold = 900;
181 static int sleepfreq = 3;
182
183 static void
184 deadlkres(void)
185 {
186 struct proc *p;
187 struct thread *td;
188 void *wchan;
189 int blkticks, i, slpticks, slptype, tryl, tticks;
(kgdb) quit
# ^D
Script done on Fri Jun 6 14:03:30 2014
Thanks.
Glen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-current/attachments/20140606/314e5806/attachment.sig>
More information about the freebsd-current
mailing list