kern/104406: [ufs] Processes get stuck
in "ufs" stateunderpersistent CPU load
Oleg Derevenetz
oleg at vsi.ru
Sun Nov 4 15:04:42 PST 2007
>>>>>> Dumpdev is swap partition on da0 (single physical disk) that
>>>>>> connected to Mylex AcceleRAID 170 RAID controller. The problem
>>>>>> arrives when I copy large amount of files from FTP to another disk
>>>>>> (da1) that is connected to the same RAID controller.
>>>>>
>>>>> If the driver or controller is misbehaving it could explain both
>>>>> problems. Any chance you can get another disk in there on a different
>>>>> controller to dump onto?
>>>>
>>>> Yes, I got IDE disk and saved kernel dump for another static hang state
>>>> on it. Here is the dump:
>>>>
>>>> ftp://oleg.vsi.ru/private/vmcore.0.zip
>>>
>>> Is this just the vmcore, or the debugging kernel also? Both are needed
>>> to make sense of the dump.
>>
>> Kernel binary with kernel config is here:
>>
>> ftp://oleg.vsi.ru/private/kernel.zip
>>
>> This kernel was built statically, and no modules loaded on boot at all.
>
> That kernel doesn't appear to match with the vmcore, are you sure it is
> the right one? Are you able to successfully run kgdb on these locally?
# kgdb kernel vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
(no debugging symbols found)...Attempt to extract a component of a value
that is not a structure pointer.
(kgdb) bt
#0 0xc0554b12 in doadump ()
#1 0xc048fe2b in db_fncall ()
#2 0xc048fc30 in db_command ()
#3 0xc048fcf8 in db_command_loop ()
#4 0xc04918f5 in db_trap ()
#5 0xc056e158 in kdb_trap ()
#6 0xc06c5960 in trap ()
#7 0xc06b176a in calltrap ()
#8 0xc056debf in kdb_enter ()
#9 0xc069bad6 in siointr1 ()
#10 0xc069b8cd in siointr ()
#11 0xc06b5565 in intr_execute_handlers ()
#12 0xc06b7cc6 in lapic_handle_intr ()
#13 0xc06b1b23 in Xapic_isr1 ()
#14 0xc06aaa4d in acpi_cpu_c1 ()
#15 0xc049b9c9 in acpi_cpu_idle ()
#16 0xc06b9e54 in cpu_idle ()
#17 0xc05402e9 in idle_proc ()
#18 0xc05400cc in fork_exit ()
#19 0xc06b17cc in fork_trampoline ()
(kgdb) info threads
114 Thread 100124 (PID=854: mc) 0xc05663ab in sched_switch ()
113 Thread 100059 (PID=851: csh) 0xc05663ab in sched_switch ()
112 Thread 100107 (PID=850: su) 0xc05663ab in sched_switch ()
111 Thread 100125 (PID=845: csh) 0xc05663ab in sched_switch ()
110 Thread 100122 (PID=844: sshd) 0xc05663ab in sched_switch ()
109 Thread 100137 (PID=840: sshd) 0xc05663ab in sched_switch ()
108 Thread 100138 (PID=839: httpd) 0xc05663ab in sched_switch ()
107 Thread 100085 (PID=838: httpd) 0xc05663ab in sched_switch ()
106 Thread 100092 (PID=837: httpd) 0xc05663ab in sched_switch ()
105 Thread 100104 (PID=836: httpd) 0xc05663ab in sched_switch ()
104 Thread 100083 (PID=835: httpd) 0xc05663ab in sched_switch ()
103 Thread 100042 (PID=834: httpd) 0xc05663ab in sched_switch ()
102 Thread 100064 (PID=833: httpd) 0xc05663ab in sched_switch ()
101 Thread 100082 (PID=832: httpd) 0xc05663ab in sched_switch ()
100 Thread 100106 (PID=831: httpd) 0xc05663ab in sched_switch ()
99 Thread 100114 (PID=830: httpd) 0xc05663ab in sched_switch ()
98 Thread 100080 (PID=829: httpd) 0xc05663ab in sched_switch ()
97 Thread 100096 (PID=828: httpd) 0xc05663ab in sched_switch ()
96 Thread 100113 (PID=827: httpd) 0xc05663ab in sched_switch ()
95 Thread 100103 (PID=826: httpd) 0xc05663ab in sched_switch ()
94 Thread 100139 (PID=825: httpd) 0xc05663ab in sched_switch ()
93 Thread 100140 (PID=824: httpd) 0xc05663ab in sched_switch ()
92 Thread 100141 (PID=823: httpd) 0xc05663ab in sched_switch ()
91 Thread 100142 (PID=822: httpd) 0xc05663ab in sched_switch ()
90 Thread 100076 (PID=821: httpd) 0xc05663ab in sched_switch ()
89 Thread 100093 (PID=820: httpd) 0xc05663ab in sched_switch ()
88 Thread 100069 (PID=819: httpd) 0xc05663ab in sched_switch ()
87 Thread 100099 (PID=818: httpd) 0xc05663ab in sched_switch ()
86 Thread 100068 (PID=817: httpd) 0xc05663ab in sched_switch ()
85 Thread 100041 (PID=816: httpd) 0xc05663ab in sched_switch ()
84 Thread 100070 (PID=815: httpd) 0xc05663ab in sched_switch ()
83 Thread 100098 (PID=814: httpd) 0xc05663ab in sched_switch ()
82 Thread 100050 (PID=813: httpd) 0xc05663ab in sched_switch ()
81 Thread 100088 (PID=812: httpd) 0xc05663ab in sched_switch ()
80 Thread 100131 (PID=811: httpd) 0xc05663ab in sched_switch ()
79 Thread 100100 (PID=810: httpd) 0xc05663ab in sched_switch ()
78 Thread 100133 (PID=809: mysqld) 0xc05663ab in sched_switch ()
77 Thread 100126 (PID=809: mysqld) 0xc05663ab in sched_switch ()
76 Thread 100134 (PID=809: mysqld) 0xc05663ab in sched_switch ()
75 Thread 100079 (PID=809: mysqld) 0xc05663ab in sched_switch ()
74 Thread 100128 (PID=809: mysqld) 0xc05663ab in sched_switch ()
73 Thread 100130 (PID=809: mysqld) 0xc05663ab in sched_switch ()
72 Thread 100039 (PID=809: mysqld) 0xc05663ab in sched_switch ()
71 Thread 100144 (PID=804: getty) 0xc05663ab in sched_switch ()
70 Thread 100145 (PID=803: getty) 0xc05663ab in sched_switch ()
69 Thread 100146 (PID=802: getty) 0xc05663ab in sched_switch ()
68 Thread 100147 (PID=801: getty) 0xc05663ab in sched_switch ()
67 Thread 100148 (PID=800: getty) 0xc05663ab in sched_switch ()
66 Thread 100149 (PID=799: getty) 0xc05663ab in sched_switch ()
65 Thread 100123 (PID=798: getty) 0xc05663ab in sched_switch ()
64 Thread 100110 (PID=797: getty) 0xc05663ab in sched_switch ()
63 Thread 100054 (PID=796: getty) 0xc05663ab in sched_switch ()
62 Thread 100111 (PID=782: inetd) 0xc05663ab in sched_switch ()
61 Thread 100112 (PID=762: httpd) 0xc05663ab in sched_switch ()
60 Thread 100077 (PID=731: sh) 0xc05663ab in sched_switch ()
59 Thread 100075 (PID=728: cron) 0xc05663ab in sched_switch ()
58 Thread 100040 (PID=720: snmpd) 0xc05663ab in sched_switch ()
57 Thread 100078 (PID=705: cron) 0xc05663ab in sched_switch ()
56 Thread 100061 (PID=699: sendmail) 0xc05663ab in sched_switch ()
55 Thread 100101 (PID=693: sendmail) 0xc05663ab in sched_switch ()
54 Thread 100095 (PID=688: sshd) 0xc05663ab in sched_switch ()
53 Thread 100119 (PID=663: ntpd) 0xc05663ab in sched_switch ()
52 Thread 100109 (PID=632: usbd) 0xc05663ab in sched_switch ()
51 Thread 100073 (PID=585: accounting) 0xc05663ab in sched_switch ()
50 Thread 100065 (PID=568: named) 0xc05663ab in sched_switch ()
49 Thread 100108 (PID=548: syslogd) 0xc05663ab in sched_switch ()
48 Thread 100121 (PID=531: ipmon) 0xc05663ab in sched_switch ()
47 Thread 100071 (PID=435: devd) 0xc05663ab in sched_switch ()
46 Thread 100089 (PID=164: adjkerntz) 0xc05663ab in sched_switch ()
45 Thread 100044 (PID=44: schedcpu) 0xc05663ab in sched_switch ()
---Type <return> to continue, or q <return> to quit---
44 Thread 100045 (PID=43: softdepflush) 0xc05663ab in sched_switch ()
43 Thread 100046 (PID=42: vnlru) 0xc05663ab in sched_switch ()
42 Thread 100047 (PID=41: syncer) 0xc05663ab in sched_switch ()
41 Thread 100048 (PID=40: bufdaemon) 0xc05663ab in sched_switch ()
40 Thread 100049 (PID=39: pagezero) 0xc05663ab in sched_switch ()
39 Thread 100027 (PID=38: vmdaemon) 0xc05663ab in sched_switch ()
38 Thread 100028 (PID=37: pagedaemon) 0xc05663ab in sched_switch ()
37 Thread 100029 (PID=36: irq7: ppc0) 0xc06b17c4 in fork_trampoline ()
36 Thread 100030 (PID=35: fdc0) 0xc05663ab in sched_switch ()
35 Thread 100031 (PID=34: swi0: sio) 0xc05663ab in sched_switch ()
34 Thread 100032 (PID=33: irq1: atkbd0) 0xc05663ab in sched_switch ()
33 Thread 100033 (PID=32: irq22: fxp0) 0xc05663ab in sched_switch ()
32 Thread 100034 (PID=31: irq20: mly0) 0xc05663ab in sched_switch ()
31 Thread 100035 (PID=30: irq15: ata1) 0xc06b17c4 in fork_trampoline ()
30 Thread 100036 (PID=29: irq14: ata0) 0xc05663ab in sched_switch ()
29 Thread 100037 (PID=28: usb1) 0xc05663ab in sched_switch ()
28 Thread 100017 (PID=27: irq17: ehci0) 0xc06b17c4 in fork_trampoline ()
27 Thread 100018 (PID=26: usbtask) 0xc05663ab in sched_switch ()
26 Thread 100019 (PID=25: usb0) 0xc05663ab in sched_switch ()
25 Thread 100020 (PID=24: irq16: ohci0) 0xc06b17c4 in fork_trampoline ()
24 Thread 100021 (PID=23: irq9: acpi0) 0xc06b17c4 in fork_trampoline ()
23 Thread 100022 (PID=22: swi5: +) 0xc06b17c4 in fork_trampoline ()
22 Thread 100023 (PID=21: thread taskq) 0xc05663ab in sched_switch ()
21 Thread 100024 (PID=20: swi6: Giant taskq) 0xc05663ab in sched_switch
()
20 Thread 100025 (PID=19: swi6: task queue) 0xc05663ab in sched_switch ()
19 Thread 100026 (PID=9: kqueue taskq) 0xc05663ab in sched_switch ()
18 Thread 100008 (PID=8: acpi_task_2) 0xc05663ab in sched_switch ()
17 Thread 100009 (PID=7: acpi_task_1) 0xc05663ab in sched_switch ()
16 Thread 100010 (PID=6: acpi_task_0) 0xc05663ab in sched_switch ()
15 Thread 100011 (PID=18: swi2: cambio) 0xc05663ab in sched_switch ()
14 Thread 100012 (PID=5: xpt_thrd) 0xc05663ab in sched_switch ()
13 Thread 100013 (PID=17: yarrow) 0xc05663ab in sched_switch ()
12 Thread 100014 (PID=4: g_down) 0xc05663ab in sched_switch ()
11 Thread 100015 (PID=3: g_up) 0xc05663ab in sched_switch ()
10 Thread 100016 (PID=2: g_event) 0xc05663ab in sched_switch ()
9 Thread 100000 (PID=16: swi3: vm) 0xc06b17c4 in fork_trampoline ()
8 Thread 100001 (PID=15: swi4: clock sio) 0xc05663ab in sched_switch ()
7 Thread 100002 (PID=14: swi1: net) 0xc05663ab in sched_switch ()
6 Thread 100003 (PID=13: idle: cpu0) 0xc06b1eb1 in Xcpustop ()
* 5 Thread 100004 (PID=12: idle: cpu1) 0xc0554b12 in doadump ()
4 Thread 100005 (PID=11: idle: cpu2) 0xc06b1eb1 in Xcpustop ()
3 Thread 100006 (PID=10: idle: cpu3) 0xc06b1eb1 in Xcpustop ()
2 Thread 100007 (PID=1: init) 0xc05663ab in sched_switch ()
1 Thread 0 (PID=0: swapper) 0xc05663ab in sched_switch ()
(kgdb) thread 26
[Switching to thread 26 (Thread 100019)]#0 0xc05663ab in sched_switch ()
(kgdb) bt
#0 0xc05663ab in sched_switch ()
#1 0xc055b868 in mi_switch ()
#2 0xc0573bd9 in sleepq_switch ()
#3 0xc0573de2 in sleepq_timedwait ()
#4 0xc055b269 in msleep ()
#5 0xc050cfa8 in usb_event_thread ()
#6 0xc05400cc in fork_exit ()
#7 0xc06b17cc in fork_trampoline ()
Anyway, kernel in kernel.zip is exactly the same kernel for which this
vmcore generated. I have no any other kernel on this machine :-)
--
Oleg Derevenetz <oleg at vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax: +7 4732 531415 http://www.vsi.ru
CenterTelecom Voronezh ISP http://isp.vsi.ru
More information about the freebsd-stable
mailing list