Recurring problem: processes block accessing UFS file system
Denis Shaposhnikov
dsh at vlink.ru
Wed Jan 11 07:02:46 PST 2006
Hi!
>>>>> "Don" == Don Lewis <truckman at FreeBSD.org> writes:
Don> Are you using any unusual file systems, such as nullfs or
Don> unionfs?
>> Yes, I'm use a lots of nullfs. This is a host system for about 20
>> jails with nullfs mounted ro system:
Don> That would be my guess as to the cause of the problem. Hopefully
Don> DEBUG_VFS_LOCKS will help pinpoint the bug.
I've got the problem again. Now I have debug kernel and crash
dump. That is an output from the kdb. Do you need any additional
information which I can get from the crash dump?
Script started on Wed Jan 11 16:19:51 2006
Connected
~KDB: enter: Line break on console
[thread pid 10 tid 100006 ]
Stopped at kdb_enter+0x32: leave
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
33279 c8fb6684 0 1 1 0000000 [SLPQ ufs 0xc690d454][SLP] sh
33017 c7f998b0 1008 1 2692 0000000 [SLPQ ufs 0xc690d300][SLP] httpd
33016 c9261000 1008 1 2692 0004000 [SLPQ ufs 0xc6bed454][SLP] parser3.cgi
33015 c8e1fd08 1008 1 2692 0004000 [SLPQ ufs 0xc6bed454][SLP] perl5.8.7
33013 c9796458 1008 1 2692 0004000 [SLPQ ufs 0xc690d454][SLP] parser3.cgi
30627 c8f2fd08 1012 21055 30627 0004002 [SLPQ ufs 0xc690d454][SLP] perl5.8.7
30210 c86e3684 1010 1 30210 0004000 [SLPQ ufs 0xc690d454][SLP] sftp-server
21225 c858522c 1012 1 21225 0004000 [SLPQ ufs 0xc6bf9300][SLP] sftp-server
21055 c6c6d684 1012 1 21055 0015002 [SLPQ ufs 0xc690d454][SLP] zsh
21031 c62978b0 1012 1 21031 0004002 [SLPQ ufs 0xc690d454][SLP] zsh
21003 c9269684 0 2728 21003 0004100 [SLPQ ufs 0xc690d454][SLP] sshd
5004 c941eadc 1010 1 5004 0004000 [SLPQ ufs 0xc690d454][SLP] sftp-server
2735 c8007684 0 1 2735 0000000 [SLPQ ufs 0xc690d454][SLP] cron
2728 c7ca1000 0 1 2728 0000100 [SLPQ ufs 0xc690d454][SLP] sshd
2705 c8006d08 1008 1 2703 0004002 [SLPQ ufs 0xc690d454][SLP] perl5.8.7
2639 c8006adc 0 1 2639 0000000 [SLPQ ufs 0xc690d454][SLP] syslogd
546 c6a10684 80 1 540 0000100 [SLPQ ufs 0xc6c07c4c][SLP] nginx
36 c60d722c 0 0 0 0000204 [SLPQ - 0xe4c45cf0][SLP] schedcpu
35 c60d7458 0 0 0 0000204 [SLPQ ktsusp 0xc60d74e0][SLP] syncer
34 c60d7684 0 0 0 0000204 [SLPQ ktsusp 0xc60d770c][SLP] vnlru
33 c60d78b0 0 0 0 0000204 [SLPQ ktsusp 0xc60d7938][SLP] bufdaemon
32 c60d7adc 0 0 0 000020c [SLPQ pgzero 0xc06d0b14][SLP] pagezero
31 c60d7d08 0 0 0 0000204 [SLPQ psleep 0xc06c8a48][SLP] vmdaemon
30 c60d8000 0 0 0 0000204 [SLPQ psleep 0xc06c8a08][SLP] pagedaemon
29 c60d822c 0 0 0 0000204 [IWAIT] irq1: atkbd0
28 c60d8458 0 0 0 0000204 [IWAIT] swi0: sio
27 c602d684 0 0 0 0000204 [IWAIT] irq18: atapci1
26 c602d8b0 0 0 0 0000204 [IWAIT] irq15: ata1
25 c602dadc 0 0 0 0000204 [IWAIT] irq14: ata0
24 c602dd08 0 0 0 0000204 [IWAIT] irq27: em1
23 c6070000 0 0 0 0000204 [IWAIT] irq26: em0
22 c607022c 0 0 0 0000204 [IWAIT] irq9: acpi0
21 c6070458 0 0 0 0000204 [IWAIT] swi2: cambio
20 c6070684 0 0 0 0000204 [IWAIT] swi6: task queue
9 c60708b0 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task2
8 c6070adc 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task1
7 c5fd822c 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task0
19 c5fd8458 0 0 0 0000204 [IWAIT] swi6: Giant taskq
6 c5fd8684 0 0 0 0000204 [SLPQ - 0xc6065000][SLP] thread taskq
18 c5fd88b0 0 0 0 0000204 [IWAIT] swi5: Fast taskq
5 c5fd8adc 0 0 0 0000204 [SLPQ - 0xc60650c0][SLP] kqueue taskq
17 c5fd8d08 0 0 0 0000204 [SLPQ - 0xc06b8460][SLP] yarrow
4 c602d000 0 0 0 0000204 [SLPQ - 0xc06b8d00][SLP] g_down
3 c602d22c 0 0 0 0000204 [SLPQ - 0xc06b8cfc][SLP] g_up
2 c602d458 0 0 0 0000204 [SLPQ - 0xc06b8cf4][SLP] g_event
16 c5fd3000 0 0 0 0000204 [IWAIT] swi1: net
15 c5fd322c 0 0 0 0000204 [IWAIT] swi3: vm
14 c5fd3458 0 0 0 000020c [IWAIT] swi4: clock sio
13 c5fd3684 0 0 0 000020c [CPU 0] idle: cpu0
12 c5fd38b0 0 0 0 000020c [CPU 1] idle: cpu1
11 c5fd3adc 0 0 0 000020c [CPU 2] idle: cpu2
10 c5fd3d08 0 0 0 000020c [CPU 3] idle: cpu3
1 c5fd8000 0 0 1 0004200 [SLPQ ufs 0xc690d5a8][SLP] init
0 c06b8e00 0 0 0 0000200 [IWAIT] swapper
21030 c8e1f8b0 1012 21003 21003 0002100 zomb[INACTIVE] sshd
5005 c93f8684 0 2728 5005 0006100 zomb[INACTIVE] sshd
5000 c83d1d08 0 2728 5000 0006100 zomb[INACTIVE] sshd
30211 c7907d08 0 2728 30211 0006100 zomb[INACTIVE] sshd
4997 c6a10adc 0 2728 4997 0006100 zomb[INACTIVE] sshd
6404 c8558684 0 2728 6404 0006100 zomb[INACTIVE] sshd
db> show lockedvnods
Locked vnodes
0xc690d550: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc9413934 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc8f84000 (pid 33279) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc054a320 at vget+0x87
#7 0xc053a1f0 at cache_lookup+0xde
#8 0xc053ac93 at vfs_cache_lookup+0xad
#9 0xc0652eaf at VOP_LOOKUP_APV+0x3b
#10 0xc053f786 at lookup+0x419
#11 0xc0540262 at namei+0x2e7
#12 0xc0558bb2 at vn_open_cred+0x3d
#13 0xc055918a at vn_open+0x33
#14 0xc054e6d3 at kern_open+0xd2
#15 0xc054f055 at open+0x36
#16 0xc0648a47 at syscall+0x23a
#17 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072602, on dev ad4s1g
0xc690d3fc: tag ufs, type VDIR
usecount 48, writecount 0, refcount 49 mountedhere 0
flags ()
v_object 0xc810c6c8 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc7f9c000 (pid 33017) with 12 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc053f430 at lookup+0xc3
#7 0xc0540262 at namei+0x2e7
#8 0xc0558d37 at vn_open_cred+0x1c2
#9 0xc055918a at vn_open+0x33
#10 0xc054e6d3 at kern_open+0xd2
#11 0xc054f055 at open+0x36
#12 0xc0648a47 at syscall+0x23a
#13 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072603, on dev ad4s1g
0xc690d2a8: tag ufs, type VDIR
usecount 2, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc839a8b8 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc8584300 (pid 21225) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc054a320 at vget+0x87
#7 0xc053a1f0 at cache_lookup+0xde
#8 0xc053ac93 at vfs_cache_lookup+0xad
#9 0xc0652eaf at VOP_LOOKUP_APV+0x3b
#10 0xc053f786 at lookup+0x419
#11 0xc0540262 at namei+0x2e7
#12 0xc0558d37 at vn_open_cred+0x1c2
#13 0xc055918a at vn_open+0x33
#14 0xc054e6d3 at kern_open+0xd2
#15 0xc054f055 at open+0x36
#16 0xc0648a47 at syscall+0x23a
#17 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072615, on dev ad4s1g
0xc6bf92a8: tag ufs, type VDIR
usecount 2, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc8a18174 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc9429780 (pid 33015) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc054a320 at vget+0x87
#7 0xc053a1f0 at cache_lookup+0xde
#8 0xc053ac93 at vfs_cache_lookup+0xad
#9 0xc0652eaf at VOP_LOOKUP_APV+0x3b
#10 0xc053f786 at lookup+0x419
#11 0xc0540262 at namei+0x2e7
#12 0xc05513e6 at kern_stat+0x3d
#13 0xc05514e8 at stat+0x2f
#14 0xc0648a47 at syscall+0x23a
#15 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072680, on dev ad4s1g
0xc6bed3fc: tag ufs, type VDIR
usecount 3, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xc84fce0c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc69d1d80 (pid 546) with 2 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc054a320 at vget+0x87
#7 0xc053a1f0 at cache_lookup+0xde
#8 0xc053ac93 at vfs_cache_lookup+0xad
#9 0xc0652eaf at VOP_LOOKUP_APV+0x3b
#10 0xc053f786 at lookup+0x419
#11 0xc0540262 at namei+0x2e7
#12 0xc0558d37 at vn_open_cred+0x1c2
#13 0xc055918a at vn_open+0x33
#14 0xc054e6d3 at kern_open+0xd2
#15 0xc054f055 at open+0x36
#16 0xc0648a47 at syscall+0x23a
#17 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072795, on dev ad4s1g
0xc6c07bf4: tag ufs, type VDIR
usecount 2, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc940a744 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc9146c00 (pid 33016) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6
#1 0xc053d780 at vop_stdlock+0x32
#2 0xc0655075 at VOP_LOCK_APV+0x3b
#3 0xc05db299 at ffs_lock+0x19
#4 0xc0655075 at VOP_LOCK_APV+0x3b
#5 0xc0557513 at vn_lock+0x6c
#6 0xc054a320 at vget+0x87
#7 0xc053a1f0 at cache_lookup+0xde
#8 0xc053ac93 at vfs_cache_lookup+0xad
#9 0xc0652eaf at VOP_LOOKUP_APV+0x3b
#10 0xc053f786 at lookup+0x419
#11 0xc0540262 at namei+0x2e7
#12 0xc0554134 at kern_rmdir+0x98
#13 0xc055436e at rmdir+0x22
#14 0xc0648a47 at syscall+0x23a
#15 0xc06326ff at Xint0x80_syscall+0x1f
ino 2072767, on dev ad4s1g
db> show lockedbufs
db>
db> alltrace
Tracing command sh pid 33279 tid 100567 td 0xc8f84000
sched_switch(c8f84000,0,1,3ef33c46,367ee305) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e91b8670,c04f425b) at mi_switch+0x150
sleepq_switch(e91b86a4,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,1) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,c0655075,c05db299,c0655075,c0557513) at acquire+0x7a
lockmgr(c690d454,2002,c690d4c4,c8f84000,e91b878c) at lockmgr+0x4ce
vop_stdlock(e91b87e4,c06a4f20,c06a4f20,e91b87e4,e91b879c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e91b87e4,e91b87b4,c0655075,e91b87e4) at VOP_LOCK_APV+0x3b
ffs_lock(e91b87e4,c0673faf,2,c690d3fc,e91b8800) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e91b87e4,e91b8814,c0504340,e91b8814) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,2002,c8f84000,c05e52a9,c690d550) at vn_lock+0x6c
vget(c690d3fc,2002,c8f84000,0,e91b88a8) at vget+0x87
cache_lookup(c690d550,e91b8b80,e91b8b94,c8f84000,c5fd1780) at cache_lookup+0xde
vfs_cache_lookup(e91b8940,e91b88ec,c690d550,e91b8b94,e91b895c) at vfs_cache_lookup+0xad
VOP_LOOKUP_APV(c06a4f20,e91b8940,e91b892c,c8f84000,2) at VOP_LOOKUP_APV+0x3b
lookup(e91b8b6c,c8d8b800,400,e91b8b88,c053d7b7) at lookup+0x419
namei(e91b8b6c,c678a618,c8f84000,e91b89f8,c0655118) at namei+0x2e7
vn_open_cred(e91b8b6c,e91b8c6c,1a4,c5fd1780,3) at vn_open_cred+0x3d
vn_open(e91b8b6c,e91b8c6c,1a4,3,0) at vn_open+0x33
kern_open(c8f84000,8084768,0,20a,1b6) at kern_open+0xd2
open(c8f84000,e91b8d04,c,3,e91b8d38) at open+0x36
syscall(3b,3b,3b,8084768,1) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a6d6b, esp = 0xbfbfd08c, ebp = 0xbfbfd128 ---
Tracing command httpd pid 33017 tid 100300 td 0xc7f9c000
sched_switch(c7f9c000,0,1,9d63b87a,bfac35b1) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8d94670,c04f425b) at mi_switch+0x150
sleepq_switch(e8d946a4,c04e4cc6,c690d300,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d300,50,c0672a31,0,d) at sleepq_wait+0xb
msleep(c690d300,c06b9cd0,50,c0672a31,0) at msleep+0x454
acquire(60000,c7f9c000,e8d94718,c064191c,dab8b000) at acquire+0x7a
lockmgr(c690d300,2002,c690d370,c7f9c000,e8d9478c) at lockmgr+0x4ce
vop_stdlock(e8d947e4,da08b520,c06a4f20,e8d947e4,e8d9479c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8d947e4,e8d947b4,c0655075,e8d947e4) at VOP_LOCK_APV+0x3b
ffs_lock(e8d947e4,4c,2,c690d2a8,e8d94800) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8d947e4,40,e8d94848,c04cd7ad) at VOP_LOCK_APV+0x3b
vn_lock(c690d2a8,2002,c7f9c000,c05e52a9,c690d3fc) at vn_lock+0x6c
vget(c690d2a8,2002,c7f9c000,0,e8d948a8) at vget+0x87
cache_lookup(c690d3fc,e8d94b80,e8d94b94,c7f9c000,c7fac400) at cache_lookup+0xde
vfs_cache_lookup(e8d94940,e8d948ec,c690d3fc,e8d94b94,e8d9495c) at vfs_cache_lookup+0xad
VOP_LOOKUP_APV(c06a4f20,e8d94940,c7f9c000,c9120050,2) at VOP_LOOKUP_APV+0x3b
lookup(e8d94b6c,c9ac3c00,400,e8d94b88,c802407c) at lookup+0x419
namei(e8d94b6c,e8d949f4,c06316b2,e8d94a18,c06326aa) at namei+0x2e7
vn_open_cred(e8d94b6c,e8d94c6c,0,c7fac400,a) at vn_open_cred+0x1c2
vn_open(e8d94b6c,e8d94c6c,0,a,0) at vn_open+0x33
kern_open(c7f9c000,81cb068,0,2,0) at kern_open+0xd2
open(c7f9c000,e8d94d04,c,0,e8d94d38) at open+0x36
syscall(811003b,e4d003b,bfbf003b,8103358,1) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0xe6faf6b, esp = 0xbfbfea7c, ebp = 0xbfbfeaa8 ---
Tracing command parser3.cgi pid 33016 tid 100652 td 0xc9146c00
sched_switch(c9146c00,0,1,5e57bc72,aaef377f) at sched_switch+0xd8
mi_switch(1,0,0,e93d86c4,c04f425b) at mi_switch+0x150
sleepq_switch(e93d86f8,c04e4cc6,c6bed454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c6bed454,50,c0672a31,0,c602a080) at sleepq_wait+0xb
msleep(c6bed454,c06b9c98,50,c0672a31,0) at msleep+0x454
acquire(60000,da1dcb4c,c0504340,da1dcb4c,4c) at acquire+0x7a
lockmgr(c6bed454,2002,c6bed4c4,c9146c00,e93d87e0) at lockmgr+0x4ce
vop_stdlock(e93d8838,120,c06a4f20,e93d8838,e93d87f0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e93d8838,e93d8808,c0655075,e93d8838) at VOP_LOCK_APV+0x3b
ffs_lock(e93d8838,f,2,c6bed3fc,e93d8854) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e93d8838,c0653a6a,c053acd2,c0652eaf) at VOP_LOCK_APV+0x3b
vn_lock(c6bed3fc,2002,c9146c00,c06326ff,6) at vn_lock+0x6c
vget(c6bed3fc,2002,c9146c00,c0653a6a,c053acd2) at vget+0x87
vfs_hash_get(c6268c00,1fa0db,2,c9146c00,e93d89bc) at vfs_hash_get+0xf5
ffs_vget(c6268c00,1fa0db,2,e93d89bc,e93d89c0) at ffs_vget+0x49
ufs_lookup(e93d8a6c,c0685ac5,c6c07bf4,e93d8c34,e93d8aa8) at ufs_lookup+0x965
VOP_CACHEDLOOKUP_APV(c06a4f20,e93d8a6c,e93d8c34,c9146c00,c7fac400) at VOP_CACHEDLOOKUP_APV+0x59
vfs_cache_lookup(e93d8b14,e93d8ac0,c6c07bf4,e93d8c34,e93d8b30) at vfs_cache_lookup+0xec
VOP_LOOKUP_APV(c06a4f20,e93d8b14,c9146c00,c065510a,c054a6f6) at VOP_LOOKUP_APV+0x3b
lookup(e93d8c0c,c954a800,400,e93d8c28,0) at lookup+0x419
namei(e93d8c0c,ffffffdf,2,0,c6268c00) at namei+0x2e7
kern_rmdir(c9146c00,824b500,0,e93d8d30,c0648a47) at kern_rmdir+0x98
rmdir(c9146c00,e93d8d04,4,d,e93d8d38) at rmdir+0x22
syscall(804003b,e50003b,bfbf003b,8310560,8310560) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (137, FreeBSD ELF32, rmdir), eip = 0xe8201bf, esp = 0xbfbfbcfc, ebp = 0xbfbfbd18 ---
Tracing command perl5.8.7 pid 33015 tid 100540 td 0xc9429780
sched_switch(c9429780,0,1,3e4e9c0e,ccaac762) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e90837c8,c04f425b) at mi_switch+0x150
sleepq_switch(e90837fc,c04e4cc6,c6bed454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c6bed454,50,c0672a31,0,c06326ff) at sleepq_wait+0xb
msleep(c6bed454,c06b9c98,50,c0672a31,0) at msleep+0x454
acquire(60000,c0648a47,c06326ff,0,0) at acquire+0x7a
lockmgr(c6bed454,2002,c6bed4c4,c9429780,e90838e4) at lockmgr+0x4ce
vop_stdlock(e908393c,c06a4f20,c06a4f20,e908393c,e90838f4) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e908393c,e908390c,c0655075,e908393c) at VOP_LOCK_APV+0x3b
ffs_lock(e908393c,c0673faf,2,c6bed3fc,e9083958) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e908393c,e908396c,c0504340,e908396c) at VOP_LOCK_APV+0x3b
vn_lock(c6bed3fc,2002,c9429780,c05e52a9,c6bf92a8) at vn_lock+0x6c
vget(c6bed3fc,2002,c9429780,0,e9083a00) at vget+0x87
cache_lookup(c6bf92a8,e9083bb4,e9083bc8,c9429780,c7fac400) at cache_lookup+0xde
vfs_cache_lookup(e9083a98,e9083a44,c6bf92a8,e9083bc8,e9083ab4) at vfs_cache_lookup+0xad
VOP_LOOKUP_APV(c06a4f20,e9083a98,c9429780,c053838f,da2dbca8) at VOP_LOOKUP_APV+0x3b
lookup(e9083ba0,c9890400,400,e9083bbc,c7ec7800) at lookup+0x419
namei(e9083ba0,c054aeb7,c0602f4b,e9083b74,c97a71f0) at namei+0x2e7
kern_stat(c9429780,86e80d4,0,e9083c14,c7d04494) at kern_stat+0x3d
stat(c9429780,e9083d04,8,5,e9083d38) at stat+0x2f
syscall(bfbf003b,bfbf003b,bfbf003b,86e80d4,86e80d4) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0xe696f0b, esp = 0xbfbf159c, ebp = 0xbfbf1618 ---
Tracing command parser3.cgi pid 33013 tid 100514 td 0xc9662000
sched_switch(c9662000,0,1,ce8a74bc,fee529ff) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e910a978,c04f425b) at mi_switch+0x150
sleepq_switch(e910a9ac,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,c057cbf1,c67ebc1c,c882b700,e910aa28) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c9662000,e910aa94) at lockmgr+0x4ce
vop_stdlock(e910aaec,0,c06a4f20,e910aaec,e910aaa4) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e910aaec,e910aabc,c0655075,e910aaec) at VOP_LOCK_APV+0x3b
ffs_lock(e910aaec,c0face68,1002,c690d3fc,e910ab08) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e910aaec,0,e910ab18,0) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c9662000,c05f6b30,c0face68) at vn_lock+0x6c
lookup(e910ac08,c8d9e800,400,e910ac24,3) at lookup+0xc3
namei(e910ac08,27f,0,0,c7f74a80) at namei+0x2e7
kern_access(c9662000,8164270,0,2,e910ad30) at kern_access+0x72
access(c9662000,e910ad04,8,1,e910ad38) at access+0x29
syscall(3b,3b,3b,8164270,8185c60) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (33, FreeBSD ELF32, access), eip = 0xe882a53, esp = 0xbfbfbd4c, ebp = 0xbfbfbd98 ---
Tracing command perl5.8.7 pid 30627 tid 100547 td 0xc9429480
sched_switch(c9429480,0,1,b326927a,259d4869) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e907d8e0,c04f425b) at mi_switch+0x150
sleepq_switch(e907d914,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,c0547c81) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,4,4,e907d9cc,c053838f) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c9429480,e907d9fc) at lockmgr+0x4ce
vop_stdlock(e907da54,c06316b2,c06a4f20,e907da54,e907da0c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e907da54,e907da24,c0655075,e907da54) at VOP_LOCK_APV+0x3b
ffs_lock(e907da54,c0655075,1002,c690d3fc,e907da70) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e907da54,c053d7b7,c065510a,c054a6f6) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c9429480,c9429480,e907daa0) at vn_lock+0x6c
lookup(e907dc08,c9a55000,400,e907dc24,0) at lookup+0xc3
namei(e907dc08,c9429480,e907db80,c9429480,8d94000) at namei+0x2e7
kern_rename(c9429480,8602b40,86f7f40,0,e907dd30) at kern_rename+0x46
rename(c9429480,e907dd04,8,c06bc640,c5fd4d60) at rename+0x29
syscall(3b,895003b,bfbf003b,8602b40,8c34008) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (128, FreeBSD ELF32, rename), eip = 0xb42721f, esp = 0xbfbfeacc, ebp = 0xbfbfeaf8 ---
Tracing command sftp-server pid 30210 tid 100409 td 0xc8583480
sched_switch(c8583480,0,1,8f67a44c,cabc55b8) at sched_switch+0xd8
mi_switch(1,0,0,e8f938b4,c04f425b) at mi_switch+0x150
sleepq_switch(e8f938e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,b) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,2169,d860e58a,ddcc828d,be7ecac0) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c8583480,e8f939d0) at lockmgr+0x4ce
vop_stdlock(e8f93a28,c86e3684,c06a4f20,e8f93a28,e8f939e0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8f93a28,e8f939f8,c0655075,e8f93a28) at VOP_LOCK_APV+0x3b
ffs_lock(e8f93a28,c06bc640,1002,c690d3fc,e8f93a44) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8f93a28,c942b480,7fffffff,c86e3748) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c8583480,c8583480,c06bf160) at vn_lock+0x6c
lookup(e8f93ba0,c9ab6c00,400,e8f93bbc,e8f93b48) at lookup+0xc3
namei(e8f93ba0,ffffffff,7fffffff,c97cac94,4) at namei+0x2e7
kern_stat(c8583480,804e080,0,e8f93c14,e8f93c40) at kern_stat+0x3d
stat(c8583480,e8f93d04,8,2,e8f93d38) at stat+0x2f
syscall(3b,3b,3b,804e080,bfbea620) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0xb497f0b, esp = 0xbfbea3fc, ebp = 0xbfbea698 ---
Tracing command sftp-server pid 21225 tid 100402 td 0xc8584300
sched_switch(c8584300,0,1,fadc2abe,86dd987) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8fae670,c04f425b) at mi_switch+0x150
sleepq_switch(e8fae6a4,c04e4cc6,c6bf9300,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c6bf9300,50,c0672a31,0,b) at sleepq_wait+0xb
msleep(c6bf9300,c06ba8f4,50,c0672a31,0) at msleep+0x454
acquire(60000,0,e8fae8fc,c0632c7e,e8fae724) at acquire+0x7a
lockmgr(c6bf9300,2002,c6bf9370,c8584300,e8fae78c) at lockmgr+0x4ce
vop_stdlock(e8fae7e4,c06a4f20,c06a4f20,e8fae7e4,e8fae79c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8fae7e4,e8fae7b4,c0655075,e8fae7e4) at VOP_LOCK_APV+0x3b
ffs_lock(e8fae7e4,c0673faf,2,c6bf92a8,e8fae800) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8fae7e4,e8fae814,c0504340,e8fae814) at VOP_LOCK_APV+0x3b
vn_lock(c6bf92a8,2002,c8584300,c05e52a9,c690d2a8) at vn_lock+0x6c
vget(c6bf92a8,2002,c8584300,0,e8fae8a8) at vget+0x87
cache_lookup(c690d2a8,e8faeb80,e8faeb94,c8584300,c8836180) at cache_lookup+0xde
vfs_cache_lookup(e8fae940,e8fae8ec,c690d2a8,e8faeb94,e8fae95c) at vfs_cache_lookup+0xad
VOP_LOOKUP_APV(c06a4f20,e8fae940,c8584300,c05589ca,c0558a6d) at VOP_LOOKUP_APV+0x3b
lookup(e8faeb6c,c99c9800,400,e8faeb88,c21845d8) at lookup+0x419
namei(e8faeb6c,c7ca4d60,c06bc640,c858522c,e8fae9fc) at namei+0x2e7
vn_open_cred(e8faeb6c,e8faec6c,1a4,c8836180,5) at vn_open_cred+0x1c2
vn_open(e8faeb6c,e8faec6c,1a4,5,c80b9700) at vn_open+0x33
kern_open(c8584300,8052000,0,1,1b6) at kern_open+0xd2
open(c8584300,e8faed04,c,369e99,0) at open+0x36
syscall(804003b,804003b,bfbf003b,0,1) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0xb497f6b, esp = 0xbfbea6bc, ebp = 0xbfbfad08 ---
Tracing command zsh pid 21055 tid 100144 td 0xc6a4c480
sched_switch(c6a4c480,0,1,2ef60aaa,c21bdf61) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8aab8b4,c04f425b) at mi_switch+0x150
sleepq_switch(e8aab8e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,0,80000,0,0) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c6a4c480,e8aab9d0) at lockmgr+0x4ce
vop_stdlock(e8aaba28,c0557513,c06a4f20,e8aaba28,e8aab9e0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8aaba28,e8aab9f8,c0655075,e8aaba28) at VOP_LOCK_APV+0x3b
ffs_lock(e8aaba28,c84ab618,1002,c690d3fc,e8aaba44) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8aaba28,e8aaba18,46,e8aaba3c) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c6a4c480,8,28) at vn_lock+0x6c
lookup(e8aabba0,c9a59000,400,e8aabbbc,c8683c8c) at lookup+0xc3
namei(e8aabba0,0,e8aabc04,e8aabbe7,e8aabbe8) at namei+0x2e7
kern_stat(c6a4c480,8054410,0,e8aabc14,0) at kern_stat+0x3d
stat(c6a4c480,e8aabd04,8,3,e8aabd38) at stat+0x2f
syscall(3b,3b,3b,bfbfe354,805441a) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0xb419f0b, esp = 0xbfbfe27c, ebp = 0xbfbfe338 ---
Tracing command zsh pid 21031 tid 100053 td 0xc6294480
sched_switch(c6294480,0,1,c3ff51ac,c4792049) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e89478b4,c04f425b) at mi_switch+0x150
sleepq_switch(e89478e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,e89478e0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,0,0,c62978b0,247) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c6294480,e89479d0) at lockmgr+0x4ce
vop_stdlock(e8947a28,c0557513,c06a4f20,e8947a28,e89479e0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8947a28,e89479f8,c0655075,e8947a28) at VOP_LOCK_APV+0x3b
ffs_lock(e8947a28,c7f5b4c4,1002,c690d3fc,e8947a44) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8947a28,e8947a18,c053d7da,c6294480) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c6294480,0,0) at vn_lock+0x6c
lookup(e8947ba0,c9547400,400,e8947bbc,c8aa8500) at lookup+0xc3
namei(e8947ba0,0,e8947c04,e8947be7,e8947be8) at namei+0x2e7
kern_stat(c6294480,b2f9db0,0,e8947c14,e8947c3c) at kern_stat+0x3d
stat(c6294480,e8947d04,8,4,e8947d38) at stat+0x2f
syscall(804003b,3b,bfbf003b,b2f9db0,8057e38) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0xb419f0b, esp = 0xbfbfeb6c, ebp = 0xbfbfec58 ---
Tracing command sshd pid 21003 tid 100669 td 0xc94fd000
sched_switch(c94fd000,0,1,2131dd7e,bcba92fa) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e93a28b4,c04f425b) at mi_switch+0x150
sleepq_switch(e93a28e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,0,0,c9269684,283) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c94fd000,e93a29d0) at lockmgr+0x4ce
vop_stdlock(e93a2a28,c0557513,c06a4f20,e93a2a28,e93a29e0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e93a2a28,e93a29f8,c0655075,e93a2a28) at VOP_LOCK_APV+0x3b
ffs_lock(e93a2a28,c7f45618,1002,c690d3fc,e93a2a44) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e93a2a28,e93a2a18,c053d7da,c94fd000) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c94fd000,0,0) at vn_lock+0x6c
lookup(e93a2ba0,c9884400,400,e93a2bbc,4) at lookup+0xc3
namei(e93a2ba0,4,c0c6c468,e93a2b68,c05eea3a) at namei+0x2e7
kern_stat(c94fd000,2836ca34,0,e93a2c14,c94fd000) at kern_stat+0x3d
stat(c94fd000,e93a2d04,8,2,e93a2d38) at stat+0x2f
syscall(3b,3b,bfbf003b,0,0) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x28356f0b, esp = 0xbfbfdb9c, ebp = 0xbfbfdc88 ---
Tracing command sftp-server pid 5004 tid 100468 td 0xc856d180
sched_switch(c856d180,0,1,8eac7b52,c9668742) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8f5175c,c04f425b) at mi_switch+0x150
sleepq_switch(e8f51790,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,da13433c) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,e8f517f0,c06316b2,e8f51814,c06326ff) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c856d180,e8f51878) at lockmgr+0x4ce
vop_stdlock(e8f518d0,3,c06a4f20,e8f518d0,e8f51888) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8f518d0,e8f518a0,c0655075,e8f518d0) at VOP_LOCK_APV+0x3b
ffs_lock(e8f518d0,e8f518dc,1002,c690d3fc,e8f518ec) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8f518d0,0,1,1) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c856d180,da7a1930,9afa) at vn_lock+0x6c
lookup(e8f51b6c,c7bc9800,400,e8f51b88,c0652dda) at lookup+0xc3
namei(e8f51b6c,c8556ee0,c06bc640,c941eadc,e8f519fc) at namei+0x2e7
vn_open_cred(e8f51b6c,e8f51c6c,1a4,c8677800,5) at vn_open_cred+0x15c
vn_open(e8f51b6c,e8f51c6c,1a4,5,c8430300) at vn_open+0x33
kern_open(c856d180,8052000,0,e02,1b6) at kern_open+0xd2
open(c856d180,e8f51d04,c,c602a080,e8f51cc8) at open+0x36
syscall(3b,3b,bfbe003b,e01,3a) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0xb497f6b, esp = 0xbfbea6bc, ebp = 0xbfbfad08 ---
Tracing command cron pid 2735 tid 100313 td 0xc7f9aa80
sched_switch(c7f9aa80,0,1,a17a4cba,9e6ea105) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8d6d8b4,c04f425b) at mi_switch+0x150
sleepq_switch(e8d6d8e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,e8d6d8e0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,e8d6d948,c06316b2,e8d6d96c,c06326ff) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c7f9aa80,e8d6d9d0) at lockmgr+0x4ce
vop_stdlock(e8d6da28,c7f9aa80,c06a4f20,e8d6da28,e8d6d9e0) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8d6da28,e8d6d9f8,c0655075,e8d6da28) at VOP_LOCK_APV+0x3b
ffs_lock(e8d6da28,c80ac3fc,1002,c690d3fc,e8d6da44) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8d6da28,4c,e8d6da28,c06316b2) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c7f9aa80,c065510a,c054a6f6) at vn_lock+0x6c
lookup(e8d6dba0,c9426c00,400,e8d6dbbc,0) at lookup+0xc3
namei(e8d6dba0,e8d6db40,60,0,c7f9aa80) at namei+0x2e7
kern_stat(c7f9aa80,804e681,0,e8d6dc14,50) at kern_stat+0x3d
stat(c7f9aa80,e8d6dd04,8,0,e8d6dd38) at stat+0x2f
syscall(3b,3b,3b,bfbfedf0,3) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x28143f0b, esp = 0xbfbfe95c, ebp = 0xbfbfec68 ---
Tracing command sshd pid 2728 tid 100238 td 0xc7ac4900
sched_switch(c7ac4900,0,1,c53f3824,ce10ace9) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8bfa7d4,c04f425b) at mi_switch+0x150
sleepq_switch(e8bfa808,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,c7ac4900) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,c0504340,c7df35cc,4c,e8bfa874) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c7ac4900,e8bfa8f0) at lockmgr+0x4ce
vop_stdlock(e8bfa948,c07ca6e0,c06a4f20,e8bfa948,e8bfa900) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8bfa948,e8bfa918,c0655075,e8bfa948) at VOP_LOCK_APV+0x3b
ffs_lock(e8bfa948,c7ac4900,1002,c690d3fc,e8bfa964) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8bfa948,0,0,0) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c7ac4900,c06326aa,3041) at vn_lock+0x6c
lookup(e8bfab94,c9892400,400,e8bfabb0,c7f5b4c4) at lookup+0xc3
namei(e8bfab94,c069ba60,2,0,c7ac4900) at namei+0x2e7
unp_connect(c7ac4900,102,0,0,e8bfac20) at unp_connect+0xc9
uipc_connect(c97d4154,c7dfb100,c7ac4900) at uipc_connect+0x46
soconnect(c97d4154,c7dfb100,c7ac4900,6a,c052bf53) at soconnect+0x5d
kern_connect(c7ac4900,4,c7dfb100,c7dfb100) at kern_connect+0xd6
connect(c7ac4900,e8bfad04,c,f,e8bfad38) at connect+0x45
syscall(3b,3b,bfbf003b,bfbfc850,bfbfc852) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (98, FreeBSD ELF32, connect), eip = 0x28344527, esp = 0xbfbfc83c, ebp = 0xbfbfc8d8 ---
Tracing command perl5.8.7 pid 2705 tid 100309 td 0xc7f9b180
sched_switch(c7f9b180,0,1,444335ac,c86a47d5) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8d7975c,c04f425b) at mi_switch+0x150
sleepq_switch(e8d79790,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,c06b610c,c6782900,901f0000,c68f0714) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c7f9b180,e8d79878) at lockmgr+0x4ce
vop_stdlock(e8d798d0,0,c06a4f20,e8d798d0,e8d79888) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8d798d0,e8d798a0,c0655075,e8d798d0) at VOP_LOCK_APV+0x3b
ffs_lock(e8d798d0,c5fd5000,1002,c690d3fc,e8d798ec) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8d798d0,c5fd5000,4,bf49f752) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c7f9b180,c5fd5000,ff) at vn_lock+0x6c
lookup(e8d79b6c,c6cf4400,400,e8d79b88,c0566fef) at lookup+0xc3
namei(e8d79b6c,0,0,e8d79a28,4) at namei+0x2e7
vn_open_cred(e8d79b6c,e8d79c6c,1a4,c7f42e80,3) at vn_open_cred+0x1c2
vn_open(e8d79b6c,e8d79c6c,1a4,3,246) at vn_open+0x33
kern_open(c7f9b180,e6a7d28,0,1,1b6) at kern_open+0xd2
open(c7f9b180,e8d79d04,c,369e99,0) at open+0x36
syscall(3b,81a003b,bfbf003b,4,e6c5f20) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0xe696f6b, esp = 0xbfbfeb1c, ebp = 0xbfbfeb48 ---
Tracing command syslogd pid 2639 tid 100308 td 0xc7f9b300
sched_switch(c7f9b300,0,1,bfebc1b4,fce5836d) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8d7c950,c04f425b) at mi_switch+0x150
sleepq_switch(e8d7c984,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb
msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454
acquire(60000,1000,c13cbba8,3b,805b050) at acquire+0x7a
lockmgr(c690d454,3002,c690d4c4,c7f9b300,e8d7ca6c) at lockmgr+0x4ce
vop_stdlock(e8d7cac4,4000,c06a4f20,e8d7cac4,e8d7ca7c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8d7cac4,e8d7ca94,c0655075,e8d7cac4) at VOP_LOCK_APV+0x3b
ffs_lock(e8d7cac4,e8d7ca94,1002,c690d3fc,e8d7cae0) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8d7cac4,e8d7cac0,c06316b2,e8d7cae4) at VOP_LOCK_APV+0x3b
vn_lock(c690d3fc,1002,c7f9b300,c055856a,c05080a7) at vn_lock+0x6c
lookup(e8d7cc0c,c9ab9000,400,e8d7cc28,0) at lookup+0xc3
namei(e8d7cc0c,c06b2960,c7febd48,c7f9b300,c7f43900) at namei+0x2e7
kern_unlink(c7f9b300,804eb7b,0,e8d7cd30,c0648a47) at kern_unlink+0x9b
unlink(c7f9b300,e8d7cd04,4,0,0) at unlink+0x22
syscall(3b,3b,3b,1,f) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (10, FreeBSD ELF32, unlink), eip = 0x28141827, esp = 0xbfbfd90c, ebp = 0xbfbfd9a8 ---
Tracing command nginx pid 546 tid 100122 td 0xc69d1d80
sched_switch(c69d1d80,0,1,f002cf9e,6c43d35e) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e8a73670,c04f425b) at mi_switch+0x150
sleepq_switch(e8a736a4,c04e4cc6,c6c07c4c,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c6c07c4c,50,c0672a31,0,c044b82a) at sleepq_wait+0xb
msleep(c6c07c4c,c06ba45c,50,c0672a31,0) at msleep+0x454
acquire(60000,c0655075,c05db299,c0655075,c0557513) at acquire+0x7a
lockmgr(c6c07c4c,2002,c6c07cbc,c69d1d80,e8a7378c) at lockmgr+0x4ce
vop_stdlock(e8a737e4,c06a4f20,c06a4f20,e8a737e4,e8a7379c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e8a737e4,e8a737b4,c0655075,e8a737e4) at VOP_LOCK_APV+0x3b
ffs_lock(e8a737e4,c0673faf,2,c6c07bf4,e8a73800) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e8a737e4,e8a73814,c0504340,e8a73814) at VOP_LOCK_APV+0x3b
vn_lock(c6c07bf4,2002,c69d1d80,c05e52a9,c6bed3fc) at vn_lock+0x6c
vget(c6c07bf4,2002,c69d1d80,0,e8a738a8) at vget+0x87
cache_lookup(c6bed3fc,e8a73b80,e8a73b94,c69d1d80,c6202900) at cache_lookup+0xde
vfs_cache_lookup(e8a73940,e8a738ec,c6bed3fc,e8a73b94,e8a7395c) at vfs_cache_lookup+0xad
VOP_LOOKUP_APV(c06a4f20,e8a73940,e8a7392c,c69d1d80,0) at VOP_LOOKUP_APV+0x3b
lookup(e8a73b6c,c9630000,400,e8a73b88,c057cbf1) at lookup+0x419
namei(e8a73b6c,c0584a9f,c7ef4900,0,e8a739a8) at namei+0x2e7
vn_open_cred(e8a73b6c,e8a73c6c,1a4,c6202900,37) at vn_open_cred+0x1c2
vn_open(e8a73b6c,e8a73c6c,1a4,37,c83b0d48) at vn_open+0x33
kern_open(c69d1d80,80f0424,0,1,1a4) at kern_open+0xd2
open(c69d1d80,e8a73d04,c,c602a480,e8a73cc8) at open+0x36
syscall(80e003b,814003b,bfbf003b,80f0475,8151000) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x282f9d6b, esp = 0xbfbfea6c, ebp = 0xbfbfeb18 ---
Tracing command schedcpu pid 36 tid 100029 td 0xc5fd9d80
sched_switch(c5fd9d80,0,1,deebc4ca,f168ca11) at sched_switch+0xd8
mi_switch(1,0,0,0,e4c45c8c) at mi_switch+0x150
sleepq_switch(3e8,e4c45cc0,c04e4e23,e4c45cf0,34) at sleepq_switch+0x115
sleepq_timedwait(e4c45cf0,34,c066a1ff,0,c7ca3900) at sleepq_timedwait+0xc
msleep(e4c45cf0,0,34,c066a1ff,3e8) at msleep+0x5b1
schedcpu_thread(0,e4c45d38,ffffffff,ffffffff,ffffffff) at schedcpu_thread+0x22c
fork_exit(c04f42d5,0,e4c45d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c45d6c, ebp = 0 ---
Tracing command syncer pid 35 tid 100030 td 0xc5fd9c00
sched_switch(c5fd9c00,0,1,edd2ea48,455594e8) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4c42c2c,c04f425b) at mi_switch+0x150
sleepq_switch(e4c42c60,c04e4cc6,c60d74e0,68,c066cfe3) at sleepq_switch+0x115
sleepq_wait(c60d74e0,68,c066cfe3,0,c60d74e0) at sleepq_wait+0xb
msleep(c60d74e0,c60d74c0,68,c066cfe3,0) at msleep+0x454
kthread_suspend_check(c60d7458,0,68,c0673dfe,fa) at kthread_suspend_check+0x65
sched_sync(0,e4c42d38,ffffffff,ffffffff,ffffffff) at sched_sync+0x11f
fork_exit(c054934e,0,e4c42d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c42d6c, ebp = 0 ---
Tracing command vnlru pid 34 tid 100031 td 0xc5fd9a80
sched_switch(c5fd9a80,0,1,befcd2e8,5f176ae8) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4c3fc40,c04f425b) at mi_switch+0x150
sleepq_switch(e4c3fc74,c04e4cc6,c60d770c,68,c066cfe3) at sleepq_switch+0x115
sleepq_wait(c60d770c,68,c066cfe3,0,c60d770c) at sleepq_wait+0xb
msleep(c60d770c,c60d76ec,68,c066cfe3,0) at msleep+0x454
kthread_suspend_check(c60d7684,c06c17e0,250,c0673dcd,3e8) at kthread_suspend_check+0x65
vnlru_proc(0,e4c3fd38,ffffffff,ffffffff,ffffffff) at vnlru_proc+0x94
fork_exit(c0548aab,0,e4c3fd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c3fd6c, ebp = 0 ---
Tracing command bufdaemon pid 33 tid 100032 td 0xc5fd9900
sched_switch(c5fd9900,0,1,b29de270,bd7eb7ab) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4c3cc84,c04f425b) at mi_switch+0x150
sleepq_switch(e4c3ccb8,c04e4cc6,c60d7938,68,c066cfe3) at sleepq_switch+0x115
sleepq_wait(c60d7938,68,c066cfe3,0,c60d7938) at sleepq_wait+0xb
msleep(c60d7938,c60d7918,68,c066cfe3,0) at msleep+0x454
kthread_suspend_check(c60d78b0,c06c150c,44,c0672ce9,3e8) at kthread_suspend_check+0x65
buf_daemon(0,e4c3cd38,ffffffff,ffffffff,ffffffff) at buf_daemon+0x106
fork_exit(c0535c97,0,e4c3cd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c3cd6c, ebp = 0 ---
Tracing command pagezero pid 32 tid 100033 td 0xc5fd9780
sched_switch(c5fd9780,0,1,1aa738e4,41803987) at sched_switch+0xd8
mi_switch(1,0,4,0,e4c39ca0) at mi_switch+0x150
sleepq_switch(493e0,e4c39cd4,c04e4e23,c06d0b14,ff) at sleepq_switch+0x115
sleepq_timedwait(c06d0b14,ff,c067dae7,0,246) at sleepq_timedwait+0xc
msleep(c06d0b14,c06c89e0,2ff,c067dae7,493e0) at msleep+0x5b1
vm_pagezero(0,e4c39d38,ffffffff,ffffffff,ffffffff) at vm_pagezero+0x8e
fork_exit(c060a5a8,0,e4c39d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c39d6c, ebp = 0 ---
Tracing command vmdaemon pid 31 tid 100034 td 0xc5fd9600
sched_switch(c5fd9600,0,1,68c14d46,5bf5a05f) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
sleepq_switch(e4c36cb0,c04e4cc6,c06c8a48,68,0) at sleepq_switch+0x115
sleepq_wait(c06c8a48,68,0,0,0) at sleepq_wait+0xb
msleep(c06c8a48,0,68,c0672ce9,0) at msleep+0x454
vm_daemon(0,e4c36d38,ffffffff,ffffffff,ffffffff) at vm_daemon+0x7c
fork_exit(c0608ca7,0,e4c36d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c36d6c, ebp = 0 ---
Tracing command pagedaemon pid 30 tid 100035 td 0xc5fd9480
sched_switch(c5fd9480,0,1,16e08acc,9be9e077) at sched_switch+0xd8
mi_switch(1,0,4,0,e4c33be0) at mi_switch+0x150
sleepq_switch(1388,e4c33c14,c04e4e23,c06c8a08,44) at sleepq_switch+0x115
sleepq_timedwait(c06c8a08,44,c0672ce9,0,0) at sleepq_timedwait+0xc
msleep(c06c8a08,c06c89e0,44,c0672ce9,1388) at msleep+0x5b1
vm_pageout(0,e4c33d38,ffffffff,ffffffff,ffffffff) at vm_pageout+0x329
fork_exit(c0606ed4,0,e4c33d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c33d6c, ebp = 0 ---
Tracing command irq1: atkbd0 pid 29 tid 100036 td 0xc60d9d80
sched_switch(c60d9d80,0,1,653f5760,1aae6dca) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c60cd700,e674fd38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c60cd700,e674fd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe674fd6c, ebp = 0 ---
Tracing command swi0: sio pid 28 tid 100037 td 0xc60d9c00
sched_switch(c60d9c00,0,1,78e163aa,280d73cf) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c60cd760,e674cd38,6d206461,6c75646f,65622065) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c60cd760,e674cd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe674cd6c, ebp = 0 ---
Tracing command irq18: atapci1 pid 27 tid 100017 td 0xc5fd5900
sched_switch(c5fd5900,0,1,91f9c4ec,bbe83083) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c60bb1d0,e4c1ed38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c60bb1d0,e4c1ed38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c1ed6c, ebp = 0 ---
Tracing command irq15: ata1 pid 26 tid 100018 td 0xc5fd5780
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 25 tid 100019 td 0xc5fd5600
sched_switch(c5fd5600,0,1,dcb75946,1d08a930) at sched_switch+0xd8
mi_switch(1,0,ffffffff,ffffffff,ffffffff) at mi_switch+0x150
ithread_loop(c60b2670,e4c18d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c60b2670,e4c18d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c18d6c, ebp = 0 ---
Tracing command irq27: em1 pid 24 tid 100020 td 0xc5fd5480
fork_trampoline() at fork_trampoline
Tracing command irq26: em0 pid 23 tid 100021 td 0xc5fd5300
sched_switch(c5fd5300,0,1,10e80ad0,3614c13) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c609dc50,e4c12d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c609dc50,e4c12d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c12d6c, ebp = 0 ---
Tracing command irq9: acpi0 pid 22 tid 100022 td 0xc5fd5180
fork_trampoline() at fork_trampoline
Tracing command swi2: cambio pid 21 tid 100023 td 0xc6071780
fork_trampoline() at fork_trampoline
Tracing command swi6: task queue pid 20 tid 100024 td 0xc6071600
sched_switch(c6071600,0,1,61baa270,dfbf6a4b) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c5fb88d0,e6678d38,e8240489,ffffc5f2,5b0cc483) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c5fb88d0,e6678d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6678d6c, ebp = 0 ---
Tracing command acpi_task2 pid 9 tid 100025 td 0xc6071480
sched_switch(c6071480,0,1,d88d1c6,a349b95) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e6675ca8,c04f425b) at mi_switch+0x150
sleepq_switch(e6675cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115
sleepq_wait(c6064e00,5c,c066a1ff,0,ad7df67a) at sleepq_wait+0xb
msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454
taskqueue_thread_loop(c07cc6b4,e6675d38,ebda3651,9850f34,c11001f6) at taskqueue_thread_loop+0x57
fork_exit(c0504e77,c07cc6b4,e6675d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6675d6c, ebp = 0 ---
Tracing command acpi_task1 pid 8 tid 100026 td 0xc6071300
sched_switch(c6071300,0,1,af3a7fc6,a346356) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e6672ca8,c04f425b) at mi_switch+0x150
sleepq_switch(e6672cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115
sleepq_wait(c6064e00,5c,c066a1ff,0,77598b73) at sleepq_wait+0xb
msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454
taskqueue_thread_loop(c07cc6b4,e6672d38,c76de081,dce7e219,98eb29d) at taskqueue_thread_loop+0x57
fork_exit(c0504e77,c07cc6b4,e6672d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6672d6c, ebp = 0 ---
Tracing command acpi_task0 pid 7 tid 100008 td 0xc5fd4300
sched_switch(c5fd4300,0,1,73bafc46,a342668) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4bf4ca8,c04f425b) at mi_switch+0x150
sleepq_switch(e4bf4cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115
sleepq_wait(c6064e00,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb
msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454
taskqueue_thread_loop(c07cc6b4,e4bf4d38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57
fork_exit(c0504e77,c07cc6b4,e4bf4d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4bf4d6c, ebp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100009 td 0xc5fd4180
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 6 tid 100010 td 0xc5fd4000
sched_switch(c5fd4000,0,1,bec3185e,4c4cc167) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4beeca8,c04f425b) at mi_switch+0x150
sleepq_switch(e4beecdc,c04e4cc6,c6065000,5c,c066a1ff) at sleepq_switch+0x115
sleepq_wait(c6065000,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb
msleep(c6065000,c606501c,5c,c066a1ff,0) at msleep+0x454
taskqueue_thread_loop(c06bffd4,e4beed38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57
fork_exit(c0504e77,c06bffd4,e4beed38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4beed6c, ebp = 0 ---
Tracing command swi5: Fast taskq pid 18 tid 100011 td 0xc5fd9300
fork_trampoline() at fork_trampoline
Tracing command kqueue taskq pid 5 tid 100012 td 0xc5fd9180
sched_switch(c5fd9180,0,1,99f92146,a32a613) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4c2dca8,c04f425b) at mi_switch+0x150
sleepq_switch(e4c2dcdc,c04e4cc6,c60650c0,5c,c066a1ff) at sleepq_switch+0x115
sleepq_wait(c60650c0,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb
msleep(c60650c0,c60650dc,5c,c066a1ff,0) at msleep+0x454
taskqueue_thread_loop(c06b9580,e4c2dd38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57
fork_exit(c0504e77,c06b9580,e4c2dd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c2dd6c, ebp = 0 ---
Tracing command yarrow pid 17 tid 100013 td 0xc5fd9000
sched_switch(c5fd9000,0,1,ed90b28e,e6cc3d64) at sched_switch+0xd8
mi_switch(1,0,0,0,e4c2ac9c) at mi_switch+0x150
sleepq_switch(64,e4c2acd0,c04e4e23,c06b8460,44) at sleepq_switch+0x115
sleepq_timedwait(c06b8460,44,c066a1ff,0,6) at sleepq_timedwait+0xc
msleep(c06b8460,0,44,c066a1ff,64) at msleep+0x5b1
random_kthread(0,e4c2ad38,ffffffff,ffffffff,ffffffff) at random_kthread+0x1f0
fork_exit(c048ff8e,0,e4c2ad38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c2ad6c, ebp = 0 ---
Tracing command g_down pid 4 tid 100014 td 0xc5fd5d80
sched_switch(c5fd5d80,0,1,acb067aa,6e026c0b) at sched_switch+0xd8
mi_switch(1,0,4,0,e4c27c68) at mi_switch+0x150
sleepq_switch(64,e4c27c9c,c04e4e23,c06b8d00,4c) at sleepq_switch+0x115
sleepq_timedwait(c06b8d00,4c,c066a1ff,0,4000) at sleepq_timedwait+0xc
msleep(c06b8d00,c06b8be8,24c,c066a1ff,64) at msleep+0x5b1
g_io_schedule_down(c5fd5d80,4c,c5fd5d80,c049a065,e4c27d00) at g_io_schedule_down+0x11c
g_down_procbody(0,e4c27d38,ffffffff,ffffffff,ffffffff) at g_down_procbody+0x97
fork_exit(c049a065,0,e4c27d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c27d6c, ebp = 0 ---
Tracing command g_up pid 3 tid 100015 td 0xc5fd5c00
sched_switch(c5fd5c00,0,1,7297e676,fbc655b8) at sched_switch+0xd8
mi_switch(1,0,4,0,e4c24c88) at mi_switch+0x150
sleepq_switch(64,e4c24cbc,c04e4e23,c06b8cfc,4c) at sleepq_switch+0x115
sleepq_timedwait(c06b8cfc,4c,c066a1ff,0,c63313ac) at sleepq_timedwait+0xc
msleep(c06b8cfc,c06b8c28,24c,c066a1ff,64) at msleep+0x5b1
g_io_schedule_up(c5fd5c00,4c,c5fd5c00,c0499fb0,e4c24d00) at g_io_schedule_up+0xc2
g_up_procbody(0,e4c24d38,ffffffff,ffffffff,ffffffff) at g_up_procbody+0x97
fork_exit(c0499fb0,0,e4c24d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c24d6c, ebp = 0 ---
Tracing command g_event pid 2 tid 100016 td 0xc5fd5a80
sched_switch(c5fd5a80,0,1,41a6af52,24a002a3) at sched_switch+0xd8
mi_switch(1,0,0,0,e4c21ca8) at mi_switch+0x150
sleepq_switch(64,e4c21cdc,c04e4e23,c06b8cf4,4c) at sleepq_switch+0x115
sleepq_timedwait(c06b8cf4,4c,c066a1ff,0,0) at sleepq_timedwait+0xc
msleep(c06b8cf4,0,4c,c066a1ff,64) at msleep+0x5b1
g_event_procbody(0,e4c21d38,ffffffff,ffffffff,ffffffff) at g_event_procbody+0xd9
fork_exit(c049a11a,0,e4c21d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c21d6c, ebp = 0 ---
Tracing command swi1: net pid 16 tid 100000 td 0xc5fd5000
sched_switch(c5fd5000,0,1,b27eab48,150f90e0) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c5fb8840,e4c0cd38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c5fb8840,e4c0cd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c0cd6c, ebp = 0 ---
Tracing command swi3: vm pid 15 tid 100001 td 0xc5fd4d80
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 14 tid 100002 td 0xc5fd4c00
sched_switch(c5fd4c00,0,1,53e95da,fc43c0ce) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
ithread_loop(c5fb8860,e4c06d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7
fork_exit(c04c1b0d,c5fb8860,e4c06d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c06d6c, ebp = 0 ---
Tracing command idle: cpu0 pid 13 tid 100003 td 0xc5fd4a80
cpustop_handler(e4c03c64,c064844e,369e99,0,be1c6ca0) at cpustop_handler+0x2c
ipi_nmi_handler(369e99,0,be1c6ca0,4af,a5f) at ipi_nmi_handler+0x2a
trap(8,c04f0028,c5fd0028,c603a900,c603a918) at trap+0x30a
calltrap() at calltrap+0x5
--- trap 0x13, eip = 0xc07bdc12, esp = 0xe4c03cac, ebp = 0xe4c03cac ---
acpi_cpu_c1(caef575e,e4c03cd8,236b,e4c03cd0,c04f2fb4) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a
idle_proc(0,e4c03d38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149
fork_exit(c04c097a,0,e4c03d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c03d6c, ebp = 0 ---
Tracing command idle: cpu1 pid 12 tid 100004 td 0xc5fd4900
cpustop_handler(e4c00c64,c064844e,ffffffff,ffffffff,ffffffff) at cpustop_handler+0x2c
ipi_nmi_handler(ffffffff,ffffffff,ffffffff,ffffffff,c5fd4900) at ipi_nmi_handler+0x2a
trap(8,c04f0028,c5fd0028,c603a700,c603a718) at trap+0x30a
calltrap() at calltrap+0x5
--- trap 0x13, eip = 0xc07bdc12, esp = 0xe4c00cac, ebp = 0xe4c00cac ---
acpi_cpu_c1(93b6c83c,e4c00cd8,236c,e4c00cd0,c04f2fb4) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a
idle_proc(0,e4c00d38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149
fork_exit(c04c097a,0,e4c00d38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4c00d6c, ebp = 0 ---
Tracing command idle: cpu2 pid 11 tid 100005 td 0xc5fd4780
cpustop_handler(e4bfdc64,c064844e,ffffffff,ffffffff,ffffffff) at cpustop_handler+0x2c
ipi_nmi_handler(ffffffff,ffffffff,ffffffff,ffffffff,c5fd4780) at ipi_nmi_handler+0x2a
trap(8,c04f0028,c5fd0028,c603ab00,c603ab18) at trap+0x30a
calltrap() at calltrap+0x5
--- trap 0x13, eip = 0xc07bdc12, esp = 0xe4bfdcac, ebp = 0xe4bfdcac ---
acpi_cpu_c1(2a172b1b,e4bfdcd8,236d,e4bfdcd0,c04f2fb4) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a
idle_proc(0,e4bfdd38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149
fork_exit(c04c097a,0,e4bfdd38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4bfdd6c, ebp = 0 ---
Tracing command idle: cpu3 pid 10 tid 100006 td 0xc5fd4600
kdb_enter(c067f27b,e4bfabfc,46,c5fd4600,c60d0c00) at kdb_enter+0x32
siointr1(e4bfac24,c04f4652,c5fd4600,c5fd4600,c5fd3d08) at siointr1+0x3de
siointr(c60d0c00,0,c5fd4600,c5fd4600,0) at siointr+0x66
intr_execute_handlers(c5fcaca0,e4bfac6c,e4bfacac,c0632a63,37) at intr_execute_handlers+0xcc
lapic_handle_intr(37) at lapic_handle_intr+0x2d
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc07bdc12, esp = 0xe4bfacac, ebp = 0xe4bfacac ---
acpi_cpu_c1(0,0,0,e4bfacd0,c04f2fb4) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a
idle_proc(0,e4bfad38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149
fork_exit(c04c097a,0,e4bfad38) at fork_exit+0x62
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4bfad6c, ebp = 0 ---
Tracing command init pid 1 tid 100007 td 0xc5fd4480
sched_switch(c5fd4480,0,1,7942a4fc,85e8071) at sched_switch+0xd8
mi_switch(1,0,c0502e44,e4bf78f0,c04f425b) at mi_switch+0x150
sleepq_switch(e4bf7924,c04e4cc6,c690d5a8,50,c0672a31) at sleepq_switch+0x115
sleepq_wait(c690d5a8,50,c0672a31,0,c064191c) at sleepq_wait+0xb
msleep(c690d5a8,c06b9c98,50,c0672a31,0) at msleep+0x454
acquire(60000,e4bf796c,c06b2ca0,c690d7f8,0) at acquire+0x7a
lockmgr(c690d5a8,2002,c690d618,c5fd4480,e4bf7a0c) at lockmgr+0x4ce
vop_stdlock(e4bf7a64,c065510a,c06a4f20,e4bf7a64,e4bf7a1c) at vop_stdlock+0x32
VOP_LOCK_APV(c06a55c0,e4bf7a64,e4bf7a34,c0655075,e4bf7a64) at VOP_LOCK_APV+0x3b
ffs_lock(e4bf7a64,0,2,c690d550,e4bf7a80) at ffs_lock+0x19
VOP_LOCK_APV(c06a4f20,e4bf7a64,c0c5fa00,0,c690d6a4) at VOP_LOCK_APV+0x3b
vn_lock(c690d550,2002,c5fd4480,7ffcaf32,369e99) at vn_lock+0x6c
vflush(c6268c00,0,2,c5fd4480,4) at vflush+0x12a
ffs_flushfiles(c6268c00,2,c5fd4480,c06b2ca0,c678f7f8) at ffs_flushfiles+0x93
ffs_unmount(c6268c00,80000,c5fd4480,c5fd4480,20) at ffs_unmount+0xea
dounmount(c6268c00,80000,c5fd4480,da054aac,0) at dounmount+0x226
vfs_unmountall(c066ec26,0,0,0,0) at vfs_unmountall+0x3b
boot(c5fd4480,e4bf7d38,bfbfe95c,c5fd8000,c5fd4480) at boot+0x80c
reboot(c5fd4480,e4bf7d04,4,0,0) at reboot+0x3c
syscall(3b,3b,3b,2,bfbfeef8) at syscall+0x23a
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (55, FreeBSD ELF32, reboot), eip = 0x8050703, esp = 0xbfbfeccc, ebp = 0xbfbfeda8 ---
Tracing command swapper pid 0 tid 0 td 0xc06b9040
sched_switch(c06b9040,0,1,e7689d46,75507e12) at sched_switch+0xd8
mi_switch(1,0,0,0,0) at mi_switch+0x150
scheduler(0,0,c5fbf79c,81ec00,81e000) at scheduler+0x551
mi_startup() at mi_startup+0xb1
begin() at begin+0x2c
db> show allpcpu
Current CPU: 3
cpuid = 0
curthread = 0xc5fd4a80: pid 13 "idle: cpu0"
curpcb = 0xe4c03d90
fpcurthread = none
idlethread = 0xc5fd4a80: pid 13 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
cpuid = 1
curthread = 0xc5fd4900: pid 12 "idle: cpu1"
curpcb = 0xe4c00d90
fpcurthread = none
idlethread = 0xc5fd4900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
cpuid = 2
curthread = 0xc5fd4780: pid 11 "idle: cpu2"
curpcb = 0xe4bfdd90
fpcurthread = none
idlethread = 0xc5fd4780: pid 11 "idle: cpu2"
APIC ID = 6
currentldt = 0x50
cpuid = 3
curthread = 0xc5fd4600: pid 10 "idle: cpu3"
curpcb = 0xe4bfad90
fpcurthread = none
idlethread = 0xc5fd4600: pid 10 "idle: cpu3"
APIC ID = 7
currentldt = 0x50
db> call doadump
Dumping 2047 MB (2 chunks)
chunk 0: 1MB (154 pages) ... ok
chunk 1: 2047MB (523968 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok
Dump complete
= 0xf
db> reset
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 3
--
DSS5-RIPE DSS-RIPN 2:550/5068 at fidonet 2:550/5069 at fidonet
mailto:dsh at vlink.ru http://neva.vlink.ru/~dsh/
More information about the freebsd-stable
mailing list