panic: spin lock held too long (w/ backtrace)
Scott Swanson
scotts at inetz.com
Fri May 11 01:00:09 UTC 2007
Kris Kennaway wrote:
> On Thu, May 10, 2007 at 02:49:45PM -0600, Scott Swanson wrote:
>> Hello all,
>>
>> I have a couple dozen SuperMicro servers in production and have had
>> reoccurring crashing issues on a couple of them that are under higher
>> load.
>>
>> After managing to pry one out of production, I have been able to
>> re-create the problem fairly consistently just by looping a script that
>> tars up a large directory and writes the the file to a different
>> partition on the same raid array.
>>
>> Can anyone point me towards the next step to debug this issue? I can
>> pursue any other debugging info that may be required.
>>
>> Regards;
>> Scott Swanson
>> Inetz Media
>>
>>
>>> uname -a
>> FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue
>> May 8 13:08:22 MDT 2007
>> root at biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE i386
>>
>>
>> root at pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug
>> /var/crash/vmcore.0
>> [GDB will not be able to debug user-mode threads:
>> /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
>> GNU gdb 6.1.1 [FreeBSD]
>>
>> Unread portion of the kernel message buffer:
>> spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds
>
> What is thread 0xc95a5900 doing?
>
> Kris
>
Is this the best way to determine the action of the thread?
Regards;
Scott
(kgdb) info threads
90 Thread 100123 (PID=18303: bsdtar) 0xc0644f5b in sched_switch
(td=0xc95a5900, newtd=0xc92aad80, flags=0) at
/usr/src/sys/kern/sched_4bsd.c:973
89 Thread 100140 (PID=17447: sh) 0xc0644f5b in sched_switch
(td=0xc9a67c00, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
88 Thread 100144 (PID=17441: tcsh) 0xc0644f5b in sched_switch
(td=0xc9a67600, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
87 Thread 100125 (PID=17440: sshd) 0xc0644f5b in sched_switch
(td=0xc95a5600, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
86 Thread 100109 (PID=17437: sshd) 0xc0644f5b in sched_switch
(td=0xc9a41480, newtd=0xc9a67600, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
85 Thread 100039 (PID=898: getty) 0xc0644f5b in sched_switch
(td=0xc93e8900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
84 Thread 100067 (PID=897: getty) 0xc0644f5b in sched_switch
(td=0xc95a6d80, newtd=0xc93e8900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
83 Thread 100087 (PID=896: getty) 0xc0644f5b in sched_switch
(td=0xc9604180, newtd=0xc9903180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
82 Thread 100088 (PID=895: getty) 0xc0644f5b in sched_switch
(td=0xc9903300, newtd=0xc9832780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
81 Thread 100089 (PID=894: getty) 0xc0644f5b in sched_switch
(td=0xc9903180, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
80 Thread 100082 (PID=893: getty) 0xc0644f5b in sched_switch
(td=0xc9604900, newtd=0xc9903180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
79 Thread 100073 (PID=892: getty) 0xc0644f5b in sched_switch
(td=0xc9832780, newtd=0xc9832180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
78 Thread 100077 (PID=891: getty) 0xc0644f5b in sched_switch
(td=0xc9832180, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
77 Thread 100078 (PID=890: getty) 0xc0644f5b in sched_switch
(td=0xc9832000, newtd=0xc9604900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
76 Thread 100090 (PID=849: cron) 0xc0644f5b in sched_switch
(td=0xc9903000, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
75 Thread 100056 (PID=843: sendmail) 0xc0644f5b in sched_switch
(td=0xc9604000, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
74 Thread 100069 (PID=839: sendmail) 0xc0644f5b in sched_switch
(td=0xc95a6a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
73 Thread 100063 (PID=833: sshd) 0xc0644f5b in sched_switch
(td=0xc9603480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
72 Thread 100107 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9a41780, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
71 Thread 100106 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9a41900, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
70 Thread 100105 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9a41a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
69 Thread 100102 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc95a5c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
68 Thread 100104 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9a41c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
67 Thread 100103 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9832a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
66 Thread 100101 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9832d80, newtd=0xc9a41780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
65 Thread 100100 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9902000, newtd=0xc9a67c00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
64 Thread 100099 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9902180, newtd=0xc9a67c00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
63 Thread 100098 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9902300, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
62 Thread 100060 (PID=832: mysqld) 0xc0644f5b in sched_switch
(td=0xc9603900, newtd=0xc9a41780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
61 Thread 100059 (PID=796: sh) 0xc0644f5b in sched_switch
(td=0xc9603a80, newtd=0xc9603900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
60 Thread 100070 (PID=784: usbd) 0xc0644f5b in sched_switch
(td=0xc95a6900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
59 Thread 100076 (PID=764: ntpd) 0xc0644f5b in sched_switch
(td=0xc9832300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
58 Thread 100051 (PID=734: nrpe2) 0xc0644f5b in sched_switch
(td=0xc95a6480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
57 Thread 100074 (PID=720: nfsd) 0xc0644f5b in sched_switch
(td=0xc9832600, newtd=0xc95a5d80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
56 Thread 100061 (PID=718: nfsd) 0xc0644f5b in sched_switch
(td=0xc9603780, newtd=0xc95a6d80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
55 Thread 100057 (PID=717: nfsd) 0xc0644f5b in sched_switch
(td=0xc9603d80, newtd=0xc93e8780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
54 Thread 100038 (PID=716: nfsd) 0xc0644f5b in sched_switch
(td=0xc93e8a80, newtd=0xc93e8780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
53 Thread 100040 (PID=715: nfsd) 0xc0644f5b in sched_switch
(td=0xc93e8780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
52 Thread 100052 (PID=713: mountd) 0xc0644f5b in sched_switch
(td=0xc95a6300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
51 Thread 100075 (PID=652: rpcbind) 0xc0644f5b in sched_switch
(td=0xc9832480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
50 Thread 100071 (PID=634: syslogd) 0xc0644f5b in sched_switch
(td=0xc95a6780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
49 Thread 100050 (PID=568: devd) 0xc0644f5b in sched_switch
(td=0xc95a6600, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
48 Thread 100058 (PID=136: adjkerntz) 0xc0644f5b in sched_switch
(td=0xc9603c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
47 Thread 100042 (PID=46: schedcpu) 0xc0644f5b in sched_switch
(td=0xc93e8480, newtd=0xc92aad80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
46 Thread 100043 (PID=45: softdepflush) 0xc0644f5b in sched_switch
(td=0xc93e8300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
---Type <return> to continue, or q <return> to quit---
45 Thread 100044 (PID=44: vnlru) 0xc0644f5b in sched_switch
(td=0xc93e8180, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
* 44 Thread 100045 (PID=43: syncer) doadump () at pcpu.h:165
43 Thread 100046 (PID=42: bufdaemon) 0xc0644f5b in sched_switch
(td=0xc93abd80, newtd=0xc93ab000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
42 Thread 100047 (PID=41: pagezero) 0xc0644f5b in sched_switch
(td=0xc93abc00, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
41 Thread 100048 (PID=40: vmdaemon) 0xc0644f5b in sched_switch
(td=0xc93aba80, newtd=0xc93e8480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
40 Thread 100049 (PID=39: pagedaemon) 0xc0644f5b in sched_switch
(td=0xc93ab900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
39 Thread 100027 (PID=38: irq7: ppc0) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
38 Thread 100028 (PID=37: fdc0) 0xc0644f5b in sched_switch
(td=0xc93ab000, newtd=0xc93e8180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
37 Thread 100029 (PID=36: swi0: sio) 0xc0644f5b in sched_switch
(td=0xc92afd80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
36 Thread 100030 (PID=35: irq1: atkbd0) 0xc0644f5b in sched_switch
(td=0xc92afc00, newtd=0xc92af300, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
35 Thread 100031 (PID=34: irq15: ata1) 0xc0644f5b in sched_switch
(td=0xc92afa80, newtd=0xc92af180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
34 Thread 100032 (PID=33: irq14: ata0) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
33 Thread 100033 (PID=32: usb2) 0xc0644f5b in sched_switch
(td=0xc92af780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
32 Thread 100034 (PID=31: irq18: uhci2) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
31 Thread 100035 (PID=30: usb1) 0xc0644f5b in sched_switch
(td=0xc92af480, newtd=0xc92af780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
30 Thread 100036 (PID=29: irq19: uhci1) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
29 Thread 100037 (PID=28: usbtask) 0xc0644f5b in sched_switch
(td=0xc93e8c00, newtd=0xc92af480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
28 Thread 100017 (PID=27: usb0) 0xc0644f5b in sched_switch
(td=0xc92ab900, newtd=0xc92af480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
27 Thread 100018 (PID=26: irq16: uhci0) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
26 Thread 100019 (PID=25: irq72: asr0) 0xc0644f5b in sched_switch
(td=0xc92ab600, newtd=0xc93ab780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
25 Thread 100020 (PID=24: irq29: em1) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
24 Thread 100021 (PID=23: irq28: em0) 0xc0644f5b in sched_switch
(td=0xc92ab300, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
23 Thread 100022 (PID=22: irq9: acpi0) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
22 Thread 100023 (PID=21: swi2: cambio) 0xc0644f5b in sched_switch
(td=0xc93ab780, newtd=0xc92abc00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
21 Thread 100024 (PID=9: kqueue taskq) 0xc0644f5b in sched_switch
(td=0xc93ab600, newtd=0xc09475c0, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
20 Thread 100025 (PID=8: acpi_task_2) 0xc0644f5b in sched_switch
(td=0xc93ab480, newtd=0xc93ab600, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
19 Thread 100026 (PID=7: acpi_task_1) 0xc0644f5b in sched_switch
(td=0xc93ab300, newtd=0xc93ab480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
18 Thread 100008 (PID=6: acpi_task_0) 0xc0644f5b in sched_switch
(td=0xc92aa300, newtd=0xc93ab300, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
17 Thread 100009 (PID=20: swi5: +) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
16 Thread 100010 (PID=5: thread taskq) 0xc0644f5b in sched_switch
(td=0xc92aa000, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
15 Thread 100011 (PID=19: swi6: Giant taskq) 0xc0644f5b in
sched_switch (td=0xc92af300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
14 Thread 100012 (PID=18: swi6: task queue) 0xc0644f5b in
sched_switch (td=0xc92af180, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
13 Thread 100013 (PID=17: yarrow) 0xc0644f5b in sched_switch
(td=0xc92af000, newtd=0xc92aba80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
12 Thread 100014 (PID=4: g_down) 0xc0644f5b in sched_switch
(td=0xc92abd80, newtd=0xc93e8000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
11 Thread 100015 (PID=3: g_up) 0xc0644f5b in sched_switch
(td=0xc92abc00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
10 Thread 100016 (PID=2: g_event) 0xc0644f5b in sched_switch
(td=0xc92aba80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
9 Thread 100000 (PID=16: swi3: vm) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
8 Thread 100001 (PID=15: swi4: clock sio) 0xc0644f5b in sched_switch
(td=0xc92aad80, newtd=0xc93e8000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
7 Thread 100002 (PID=14: swi1: net) 0xc0644f5b in sched_switch
(td=0xc92aac00, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
6 Thread 100003 (PID=13: idle: cpu0) 0xc0644f5b in sched_switch
(td=0xc92aaa80, newtd=0xc92aad80, flags=2) at
/usr/src/sys/kern/sched_4bsd.c:973
5 Thread 100004 (PID=12: idle: cpu1) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
4 Thread 100005 (PID=11: idle: cpu2) 0xc0644f5b in sched_switch
(td=0xc92aa780, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
3 Thread 100006 (PID=10: idle: cpu3) fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
2 Thread 100007 (PID=1: init) 0xc0644f5b in sched_switch
(td=0xc92aa480, newtd=0xc9603480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
1 Thread 0 (PID=0: swapper) 0xc0644f5b in sched_switch
(td=0xc09475c0, newtd=0xc93ab780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
More information about the freebsd-stable
mailing list