bwrite() wdrain hang in -current
Don Lewis
truckman at FreeBSD.org
Tue May 6 21:58:44 PDT 2003
I've been working on a patch to fix the vnode locking bugs in NFS server
code in -current and stumbled across a problem that appears to be
unrelated to NFS. As a torture test, I've NFS mounted /tmp/obj on
/usr/obj and run "make buildworld". The first time I tried this, I
didn't have any problems, so I tried again with -j10, which wedged all
the nfsd processes. After updating to the latest source, I ran "make
buildworld" again and got another hang, even without the -j option.
The nfsd processes all hang while attempting to lock a vnode, but the
reason I don't think this is related to NFS is that the vnode lock is
held by bufdaemon, which is sleeping on runningbufreq.
The vnode in question this time around is 0xc9ba27fc, which corresponds
to /tmp/obj/usr/src/gnu/usr.bin/cc/cc1/cc1. The stack for the bufdaemon
thread holding the lock is:
Proc 0xc6153000 [SLPQ wdrain c05ce978][SLP] bufdaemon
mi_switch(c60d8260,44,c050c1cf,ca,1) at mi_switch+0x210
msleep(c05ce978,c05ce980,44,c0511486,0) at msleep+0x432
bwrite(d28ce2e0,0,c0511368,697,137e400) at bwrite+0x442
vfs_bio_awrite(d28ce2e0,0,c0511368,87b,0) at vfs_bio_awrite+0x221
flushbufqueues(0,0,c0511368,10e,64) at flushbufqueues+0x17d
buf_daemon(0,e0a92d48,c0509769,310,0) at buf_daemon+0xdc
fork_exit(c0363240,0,e0a92d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
What is puzzling is why this process is sleeping here. It appears that
maybe a wakeup didn't happen. This machine has 1 GB of RAM, so I don't
think memory pressure should be a cause. Here's the source at bwrite+0x442
(kgdb) list *bwrite+0x442
0xc0360f82 is in bwrite (/usr/src/sys/kern/vfs_bio.c:362).
357 waitrunningbufspace(void)
358 {
359 mtx_lock(&rbreqlock);
360 while (runningbufspace > hirunningspace) {
361 ++runningbufreq;
362 msleep(&runningbufreq, &rbreqlock, PVM, "wdrain", 0);
363 }
364 mtx_unlock(&rbreqlock);
365 }
366
which is part of the waitrunningbufspace() inline routine. Bufdaemon
shouldn't be sleeping here because
(kgdb) print runningbufspace
$1 = 548864
(kgdb) print hirunningspace
$2 = 1048576
(kgdb) print runningbufreq
$3 = 3
Once all the nfsd processes block, I get an NFS server not responding
message and the buildworld freezes, but the system is otherwise
responsive.
Here is all the stuff I thought to print from DDB:
db> ps
pid proc addr uid ppid pgrp flag stat wmesg wchan cmd
4160 c6a5db40 e6e3d000 0 4159 559 0004002 norm[SLPQ nfsrcvlk c6640e04][SLP] ld
4159 c66a73c0 e6d8a000 0 4158 559 0004002 norm[SLPQ wait c66a73c0][SLP] cc
4158 c76ad000 e6e7d000 0 4141 559 0004002 norm[SLPQ wait c76ad000][SLP] sh
4141 c76adb40 e6e87000 0 3254 559 0004002 norm[SLPQ wait c76adb40][SLP] make
3254 c66a71e0 e6d89000 0 3253 559 0004002 norm[SLPQ wait c66a71e0][SLP] sh
3253 c76abb40 e6e7b000 0 1779 559 0004002 norm[SLPQ wait c76abb40][SLP] make
1779 c76ab3c0 e6e6f000 0 1778 559 0004002 norm[SLPQ wait c76ab3c0][SLP] sh
1778 c686dd20 e6e2b000 0 1732 559 0004002 norm[SLPQ wait c686dd20][SLP] make
1732 c76ad960 e6e86000 0 1731 559 0004002 norm[SLPQ wait c76ad960][SLP] sh
1731 c76abd20 e6e7c000 0 1730 559 0004002 norm[SLPQ wait c76abd20][SLP] make
1730 c6661d20 e6d5e000 0 98448 559 0004002 norm[SLPQ wait c6661d20][SLP] sh
898 c76ab780 e6e75000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 10
897 c6a5c3c0 e6d82000 0 0 0 0000204 norm[SLPQ sbwait c6689b64][SLP] nfsiod 9
98448 c76ad780 e6e85000 0 98447 559 0004002 norm[SLPQ wait c76ad780][SLP] make
98447 c6a5d5a0 e6e3a000 0 604 559 0004002 norm[SLPQ wait c6a5d5a0][SLP] sh
97622 c66a8960 e6dbc000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 6
97621 c6a5cd20 e6d87000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 5
93797 c6a5d000 e6e37000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 4
604 c686c5a0 e6dd1000 0 602 559 0004002 norm[SLPQ wait c686c5a0][SLP] make
602 c686d000 e6dd6000 0 559 559 0004002 norm[SLPQ wait c686d000][SLP] sh
559 c66a8d20 e6dbe000 0 554 559 0004002 norm[SLPQ wait c66a8d20][SLP] make
554 c686c780 e6dd2000 0 553 554 0004002 norm[SLPQ pause e6dd2000][SLP] csh
553 c686c960 e6dd3000 1001 547 553 0004102 norm[SLPQ wait c686c960][SLP] su
547 c686d5a0 e6dd9000 1001 546 547 0004002 norm[SLPQ pause e6dd9000][SLP] csh
546 c66a83c0 e6db9000 1001 544 544 0000100 norm[CVQ select c05ce574][SLP] sshd
544 c66a81e0 e6db8000 0 416 544 0000100 norm[SLPQ sbwait c6862264][SLP] sshd
543 c66a7960 e6d8d000 0 1 543 0004002 norm[SLPQ ttyin c639de10][SLP] getty
542 c665d5a0 e6d03000 0 1 542 0004002 norm[SLPQ ttyin c66df210][SLP] getty
541 c66a85a0 e6dba000 0 1 541 0004002 norm[SLPQ ttyin c66df810][SLP] getty
540 c665d000 e6d00000 0 1 540 0004002 norm[SLPQ ttyin c66dfe10][SLP] getty
539 c665d1e0 e6d01000 0 1 539 0004002 norm[SLPQ ttyin c66e0410][SLP] getty
538 c66a7d20 e6d8f000 0 1 538 0004002 norm[SLPQ ttyin c6147410][SLP] getty
537 c6661b40 e6d5d000 0 1 537 0004002 norm[SLPQ ttyin c60bfa10][SLP] getty
536 c63a33c0 e4b00000 0 1 536 0004002 norm[SLPQ ttyin c60c0e10][SLP] getty
524 c6155b40 e4afb000 0 1 524 0000000 norm[CVQ select c05ce574][SLP] inetd
503 c66615a0 e6d5a000 0 1 503 0000000 norm[CVQ select c05ce574][SLP] moused
486 c66a7000 e6d88000 65534 1 486 0004002 norm[RUNQ] setiathome
478 c66a75a0 e6d8b000 0 1 478 0000000 norm[SLPQ nanslp c05a976c][SLP] upslog
477 c66a7780 e6d8c000 66 475 475 0000100 norm[SLPQ nanslp c05a976c][SLP] upsmon
475 c66a7b40 e6d8e000 0 1 475 0000000 norm[SLPQ piperd c63c0370][SLP] upsmon
473 c66a8000 e6d90000 66 1 473 0000101 norm[CVQ select c05ce574][SLP] upsd
468 c6661000 e6d57000 66 1 468 0000101 norm[SLPQ ttyin c60bf010][SLP] newapc
463 c665db40 e6d06000 0 1 463 0000000 norm[CVQ select c05ce574][SLP] healthd
440 c66611e0 e6d58000 0 1 440 0000000 norm[SLPQ nanslp c05a976c][SLP] cron
425 c665dd20 e6d07000 25 1 425 0000100 norm[SLPQ pause e6d07000][SLP] sendmail
422 c665d960 e6d05000 0 1 422 0000100 norm[CVQ select c05ce574][SLP] sendmail
416 c665d780 e6d04000 0 1 416 0000100 norm[CVQ select c05ce574][SLP] sshd
400 c63a3d20 e4b05000 0 1 400 0000000 norm[CVQ select c05ce574][SLP] usbd
383 c665d3c0 e6d02000 0 1 383 0000000 norm[CVQ select c05ce574][SLP] ntpd
368 c66613c0 e6d59000 0 1 368 0000000 norm[CVQ select c05ce574][SLP] lpd
339 c6661780 e6d5b000 0 334 334 0000000 norm[SLPQ ufs c9ba28b8][SLP] nfsd
338 c63a31e0 e4aff000 0 334 334 0000000 norm[SLPQ ufs c9ba28b8][SLP] nfsd
336 c61553c0 e4ad0000 0 334 334 0000000 norm[SLPQ ufs c9ba28b8][SLP] nfsd
335 c6155780 e4ad2000 0 334 334 0000000 norm[SLPQ ufs c9ba28b8][SLP] nfsd
334 c6155d20 e4afc000 0 1 334 0000000 norm[CVQ select c05ce574][SLP] nfsd
328 c61555a0 e4ad1000 0 1 328 0000000 norm[CVQ select c05ce574][SLP] mountd
266 c6155960 e4ad3000 0 1 266 0000000 norm[CVQ select c05ce574][SLP] rpcbind
250 c63a3000 e4afe000 0 1 250 0000000 norm[CVQ select c05ce574][SLP] syslogd
46 c63a35a0 e4b01000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 3
45 c63a3780 e4b02000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 2
44 c63a3960 e4b03000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 1
43 c63a3b40 e4b04000 0 0 0 0000204 norm[SLPQ nfsrcvlk c6640e04][SLP] nfsiod 0
42 c60d7b40 e0ab1000 0 0 0 0000204 norm[SLPQ vlruwt c60d7b40][SLP] vnlru
41 c60d7d20 e0ab2000 0 0 0 0000204 norm[SLPQ wdrain c05ce978][SLP] syncer
40 c6153000 e4ac6000 0 0 0 0000204 norm[SLPQ wdrain c05ce978][SLP] bufdaemon
39 c61531e0 e4ac7000 0 0 0 000020c norm[RUNQ] pagezero
9 c61533c0 e4ac8000 0 0 0 0000204 norm[SLPQ psleep c05d67f4][SLP] vmdaemon
8 c61535a0 e4ac9000 0 0 0 0000204 norm[SLPQ psleep c05d67e0][SLP] pagedaemon
38 c6153780 e4aca000 0 0 0 0000204 new [IWAIT] irq8: rtc
37 c6153960 e4acb000 0 0 0 0000204 new [IWAIT] irq0: clk
36 c6153b40 e4acc000 0 0 0 0000204 new [IWAIT] irq12: psm0
35 c6153d20 e4acd000 0 0 0 0000204 norm[CPU 0] irq1: atkbd0
34 c6155000 e4ace000 0 0 0 0000204 norm[IWAIT] irq7: ppc0
33 c61551e0 e4acf000 0 0 0 0000204 new [IWAIT] irq3: sio1
32 c60015a0 e0a7a000 0 0 0 0000204 new [IWAIT] irq4: sio0
31 c6001780 e0a7b000 0 0 0 0000204 norm[IWAIT] swi0: tty:sio
30 c6001960 e0a7c000 0 0 0 0000204 norm[IWAIT] irq6: fdc0
29 c6001b40 e0a7d000 0 0 0 0000204 norm[IWAIT] irq11: fxp0
28 c6001d20 e0a7e000 0 0 0 0000204 norm[SLPQ usbevt c612f210][SLP] usb1
27 c60d7000 e0a84000 0 0 0 0000204 norm[SLPQ usbtsk c05a0058][SLP] usbtask
26 c60d71e0 e0a85000 0 0 0 0000204 norm[SLPQ usbevt c613c210][SLP] usb0
25 c60d73c0 e0a86000 0 0 0 0000204 new [IWAIT] irq10: uhci0 uhci1
24 c60d75a0 e0a87000 0 0 0 0000204 norm[IWAIT] irq15: ata1 ahc0
--More-- 23 c60d7780 e0a88000 0 0 0 0000204 new [IWAIT] irq14: ata0
7 c60d7960 e0a89000 0 0 0 0000204 norm[SLPQ actask c06fb0cc][SLP] acpi_task2
6 c20e71e0 df048000 0 0 0 0000204 norm[SLPQ actask c06fb0cc][SLP] acpi_task1
5 c20e73c0 df049000 0 0 0 0000204 norm[SLPQ actask c06fb0cc][SLP] acpi_task0
22 c20e75a0 df04a000 0 0 0 0000204 new [IWAIT] irq9: acpi0
21 c20e7780 df04b000 0 0 0 0000204 new [IWAIT] irq13:
20 c20e7960 df04c000 0 0 0 0000204 new [IWAIT] swi6: task queue
19 c20e7b40 df04d000 0 0 0 0000204 norm[IWAIT] swi3: cambio
18 c20e7d20 df075000 0 0 0 0000204 new [IWAIT] swi2: camnet
17 c6001000 e0a77000 0 0 0 0000204 new [IWAIT] swi5:+
16 c60011e0 e0a78000 0 0 0 0000204 norm[IWAIT] swi6: acpitaskq
15 c60013c0 e0a79000 0 0 0 0000204 norm[SLPQ sleep c0593ea0][SLP] random
4 c20e0000 deff1000 0 0 0 0000204 norm[SLPQ g_down c05a25b8][SLP] g_down
3 c20e01e0 df040000 0 0 0 0000204 norm[SLPQ g_up c05a25b4][SLP] g_up
2 c20e03c0 df041000 0 0 0 0000204 norm[SLPQ g_events c05a25ac][SLP] g_event
14 c20e05a0 df042000 0 0 0 0000204 new [IWAIT] swi4: vm
13 c20e0780 df043000 0 0 0 000020c norm[RUNQ] swi7: tty:sio clock
12 c20e0960 df044000 0 0 0 0000204 norm[IWAIT] swi1: net
11 c20e0b40 df045000 0 0 0 000020c norm[Can run] idle
1 c20e0d20 df046000 0 0 1 0004200 norm[SLPQ wait c20e0d20][SLP] init
10 c20e7000 df047000 0 0 0 0000204 norm[CVQ ktrace c05a5bf4][SLP] ktrace
--More-- 0 c05a2720 c072a000 0 0 0 0000200 norm[SLPQ sched c05a2720][SLP] swapper
db> tr 335
mi_switch(c20e8390,50,c050c1cf,ca,c20e8390) at mi_switch+0x210
msleep(c9ba28b8,c05a7d08,50,c0515f0d,0) at msleep+0x432
acquire(df059930,1000040,600,ef,c20e8390) at acquire+0x9e
lockmgr(c9ba28b8,1010002,c9ba27fc,c20e8390,df05994c) at lockmgr+0x387
vop_stdlock(df059974,df059958,c0455358,df059974,df059994) at vop_stdlock+0x2c
vop_defaultop(df059974,df059994,c037feb9,df059974,246) at vop_defaultop+0x18
ufs_vnoperate(df059974,246,c051fb05,0,df059994) at ufs_vnoperate+0x18
vn_lock(c9ba27fc,10002,c20e8390,163,c05b2320) at vn_lock+0xe9
vget(c9ba27fc,10002,c20e8390,7f,c20e8390) at vget+0xd6
ufs_ihashget(c63a6b00,137b94,2,df059a90,c05ce4dc) at ufs_ihashget+0xdc
ffs_vget(c6393000,137b94,2,df059a90,0) at ffs_vget+0x3e
ufs_fhtovp(c6393000,df059b38,df059b1c,c6394000,df059c5c) at ufs_fhtovp+0x37
ffs_fhtovp(c6393000,df059b38,df059b1c,df059b2c,c05b1ea0) at ffs_fhtovp+0x47
nfsrv_commit(c6b2bc00,c6630c00,c20e8390,df059c8c,0) at nfsrv_commit+0x150
nfssvc_nfsd(c20e8390,0,c05199d2,96,df059ce0) at nfssvc_nfsd+0x3aa
nfssvc(c20e8390,df059d10,c0526328,3fb,2) at nfssvc+0x17d
syscall(2f,2f,2f,0,4) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x480b7bb3, esp = 0xbfbffa1c, ebp = 0xbfbffa38 ---
db> tr 336
mi_switch(c20e85f0,50,c050c1cf,ca,c20e85f0) at mi_switch+0x210
msleep(c9ba28b8,c05a7d08,50,c0515f0d,0) at msleep+0x432
acquire(df05f930,1000040,600,ef,c20e85f0) at acquire+0x9e
lockmgr(c9ba28b8,1010002,c9ba27fc,c20e85f0,df05f94c) at lockmgr+0x387
vop_stdlock(df05f974,df05f958,c0455358,df05f974,df05f994) at vop_stdlock+0x2c
vop_defaultop(df05f974,df05f994,c037feb9,df05f974,246) at vop_defaultop+0x18
ufs_vnoperate(df05f974,246,c051fb05,0,df05f994) at ufs_vnoperate+0x18
vn_lock(c9ba27fc,10002,c20e85f0,163,c05b2320) at vn_lock+0xe9
vget(c9ba27fc,10002,c20e85f0,7f,c20e85f0) at vget+0xd6
ufs_ihashget(c63a6b00,137b94,2,df05fa90,c05ce4dc) at ufs_ihashget+0xdc
ffs_vget(c6393000,137b94,2,df05fa90,0) at ffs_vget+0x3e
ufs_fhtovp(c6393000,df05fb38,df05fb1c,c6394c00,df05fc5c) at ufs_fhtovp+0x37
ffs_fhtovp(c6393000,df05fb38,df05fb1c,df05fb2c,c05b1ea0) at ffs_fhtovp+0x47
nfsrv_commit(c6b5c500,c6630c00,c20e85f0,df05fc8c,0) at nfsrv_commit+0x150
nfssvc_nfsd(c20e85f0,0,c05199d2,96,df05fce0) at nfssvc_nfsd+0x3aa
nfssvc(c20e85f0,df05fd10,c0526328,3fb,2) at nfssvc+0x17d
syscall(2f,2f,2f,1,4) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x480b7bb3, esp = 0xbfbffa1c, ebp = 0xbfbffa38 ---
db> tr 338
mi_switch(c61565f0,50,c050c1cf,ca,c61565f0) at mi_switch+0x210
msleep(c9ba28b8,c05a7d08,50,c0515f0d,0) at msleep+0x432
acquire(e4ae5930,1000040,600,ef,c61565f0) at acquire+0x9e
lockmgr(c9ba28b8,1010002,c9ba27fc,c61565f0,e4ae594c) at lockmgr+0x387
vop_stdlock(e4ae5974,e4ae5958,c0455358,e4ae5974,e4ae5994) at vop_stdlock+0x2c
vop_defaultop(e4ae5974,e4ae5994,c037feb9,e4ae5974,246) at vop_defaultop+0x18
ufs_vnoperate(e4ae5974,246,c051fb05,0,e4ae5994) at ufs_vnoperate+0x18
vn_lock(c9ba27fc,10002,c61565f0,163,c05b2320) at vn_lock+0xe9
vget(c9ba27fc,10002,c61565f0,7f,c61565f0) at vget+0xd6
ufs_ihashget(c63a6b00,137b94,2,e4ae5a90,c05ce4dc) at ufs_ihashget+0xdc
ffs_vget(c6393000,137b94,2,e4ae5a90,0) at ffs_vget+0x3e
ufs_fhtovp(c6393000,e4ae5b38,e4ae5b1c,c6394800,e4ae5c5c) at ufs_fhtovp+0x37
ffs_fhtovp(c6393000,e4ae5b38,e4ae5b1c,e4ae5b2c,c05b1ea0) at ffs_fhtovp+0x47
nfsrv_commit(c6b23900,c6630c00,c61565f0,e4ae5c8c,0) at nfsrv_commit+0x150
nfssvc_nfsd(c61565f0,0,c05199d2,96,e4ae5ce0) at nfssvc_nfsd+0x3aa
nfssvc(c61565f0,e4ae5d10,c0526328,3fb,2) at nfssvc+0x17d
syscall(2f,2f,2f,2,4) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x480b7bb3, esp = 0xbfbffa1c, ebp = 0xbfbffa38 ---
db> tr 339
mi_switch(c665e980,50,c050c1cf,ca,c665e980) at mi_switch+0x210
msleep(c9ba28b8,c05a7d08,50,c0515f0d,0) at msleep+0x432
acquire(e6d23930,1000040,600,ef,c665e980) at acquire+0x9e
lockmgr(c9ba28b8,1010002,c9ba27fc,c665e980,e6d2394c) at lockmgr+0x387
vop_stdlock(e6d23974,e6d23958,c0455358,e6d23974,e6d23994) at vop_stdlock+0x2c
vop_defaultop(e6d23974,e6d23994,c037feb9,e6d23974,246) at vop_defaultop+0x18
ufs_vnoperate(e6d23974,246,c051fb05,0,e6d23994) at ufs_vnoperate+0x18
vn_lock(c9ba27fc,10002,c665e980,163,c05b2320) at vn_lock+0xe9
vget(c9ba27fc,10002,c665e980,7f,c665e980) at vget+0xd6
ufs_ihashget(c63a6b00,137b94,2,e6d23a90,c05ce4dc) at ufs_ihashget+0xdc
ffs_vget(c6393000,137b94,2,e6d23a90,0) at ffs_vget+0x3e
ufs_fhtovp(c6393000,e6d23b38,e6d23b1c,c6394600,e6d23c5c) at ufs_fhtovp+0x37
ffs_fhtovp(c6393000,e6d23b38,e6d23b1c,e6d23b2c,c05b1ea0) at ffs_fhtovp+0x47
nfsrv_commit(c6b3ba00,c6630c00,c665e980,e6d23c8c,0) at nfsrv_commit+0x150
nfssvc_nfsd(c665e980,0,c05199d2,96,e6d23ce0) at nfssvc_nfsd+0x3aa
nfssvc(c665e980,e6d23d10,c0526328,3fb,2) at nfssvc+0x17d
syscall(2f,2f,2f,3,4) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x480b7bb3, esp = 0xbfbffa1c, ebp = 0xbfbffa38 ---
db> tr
Debugger(c0523a5c,4,1,0,1) at Debugger+0x54
scgetc(c20c1800,2,1d7,c05ac028,c615e440) at scgetc+0x445
sckbdevent(c05d6ea0,0,c20c1800,c05a70e0,8) at sckbdevent+0x1e0
atkbd_intr(c05d6ea0,0,df068d0c,c03093b2,c05d6ea0) at atkbd_intr+0x2c
atkbd_isa_intr(c05d6ea0,0,c05098be,218,c6153d20) at atkbd_isa_intr+0x21
ithread_loop(c615d080,df068d48,c0509769,310,0) at ithread_loop+0x182
fork_exit(c0309230,c615d080,df068d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xdf068d7c, ebp = 0 ---
db> sho lockedvnod
Locked vnodes
0xc6625b68: tag syncer, type VNON, usecount 1, writecount 0, refcount 0, lock type syncer: EXCL (count 1) by thread 0xc60d8390
0xc9ba27fc: tag ufs, type VREG, usecount 4, writecount 0, refcount 194, flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc60d8260 with 4 pending
ino 1276820, on dev da0s1a (4, 15)
0xc9ba1db0: tag ufs, type VREG, usecount 1, writecount 0, refcount 185, flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc60d8390
ino 1276823, on dev da0s1a (4, 15)
0xc9ba1c8c: tag nfs, type VREG, usecount 1, writecount 1, refcount 368, flags (VV_OBJBUF), lock type nfs: EXCL (count 1) by thread 0xc66a9980
fileid 1276823 fsid 0x500ff04
db> sho thread 0xc60d8390
Proc 0xc60d7d20 [SLPQ wdrain c05ce978][SLP] syncer
mi_switch(c60d8390,44,c050c1cf,ca,e0a95aa8) at mi_switch+0x210
msleep(c05ce978,c05ce980,44,c0511486,0) at msleep+0x432
bwrite(d2804c38,e0a95b4c,c0369751,d2804c38,d2804d68) at bwrite+0x442
bawrite(d2804c38,d2804d68,8,3c4,20020080) at bawrite+0x1c
cluster_wbuild(c9ba1db0,4000,1d,0,2) at cluster_wbuild+0x6d1
vfs_bio_awrite(d2a81e20,0,c051f4ff,d7,c0370307) at vfs_bio_awrite+0x1f9
ffs_fsync(e0a95be4,c05051b3,c057e520,c9ba1db0,c20c8e80) at ffs_fsync+0x2b2
VOP_FSYNC(c9ba1db0,c20c8e80,3,c60d8390,0) at VOP_FSYNC+0x5b
ffs_sync(c6393000,3,c20c8e80,c60d8390,c6393000) at ffs_sync+0x184
sync_fsync(e0a95c94,c05051b3,c057e520,c6625b68,c20c8e80) at sync_fsync+0x16a
VOP_FSYNC(c6625b68,c20c8e80,3,c60d8390,0) at VOP_FSYNC+0x5b
sched_sync(0,e0a95d48,c0509769,310,0) at sched_sync+0x16b
fork_exit(c0373080,0,e0a95d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xe0a95d7c, ebp = 0 ---
db> sho thread 0xc60d8260
Proc 0xc6153000 [SLPQ wdrain c05ce978][SLP] bufdaemon
mi_switch(c60d8260,44,c050c1cf,ca,1) at mi_switch+0x210
msleep(c05ce978,c05ce980,44,c0511486,0) at msleep+0x432
bwrite(d28ce2e0,0,c0511368,697,137e400) at bwrite+0x442
vfs_bio_awrite(d28ce2e0,0,c0511368,87b,0) at vfs_bio_awrite+0x221
flushbufqueues(0,0,c0511368,10e,64) at flushbufqueues+0x17d
buf_daemon(0,e0a92d48,c0509769,310,0) at buf_daemon+0xdc
fork_exit(c0363240,0,e0a92d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xe0a92d7c, ebp = 0 ---
db> sho thread 0xc60d8390
Proc 0xc60d7d20 [SLPQ wdrain c05ce978][SLP] syncer
mi_switch(c60d8390,44,c050c1cf,ca,e0a95aa8) at mi_switch+0x210
msleep(c05ce978,c05ce980,44,c0511486,0) at msleep+0x432
bwrite(d2804c38,e0a95b4c,c0369751,d2804c38,d2804d68) at bwrite+0x442
bawrite(d2804c38,d2804d68,8,3c4,20020080) at bawrite+0x1c
cluster_wbuild(c9ba1db0,4000,1d,0,2) at cluster_wbuild+0x6d1
vfs_bio_awrite(d2a81e20,0,c051f4ff,d7,c0370307) at vfs_bio_awrite+0x1f9
ffs_fsync(e0a95be4,c05051b3,c057e520,c9ba1db0,c20c8e80) at ffs_fsync+0x2b2
VOP_FSYNC(c9ba1db0,c20c8e80,3,c60d8390,0) at VOP_FSYNC+0x5b
ffs_sync(c6393000,3,c20c8e80,c60d8390,c6393000) at ffs_sync+0x184
sync_fsync(e0a95c94,c05051b3,c057e520,c6625b68,c20c8e80) at sync_fsync+0x16a
VOP_FSYNC(c6625b68,c20c8e80,3,c60d8390,0) at VOP_FSYNC+0x5b
sched_sync(0,e0a95d48,c0509769,310,0) at sched_sync+0x16b
fork_exit(c0373080,0,e0a95d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xe0a95d7c, ebp = 0 ---
db> sho thread 0xc66a9980
Proc 0xc6a5db40 [SLPQ nfsrcvlk c6640e04][SLP] ld
mi_switch(c66a9980,53,c050c1cf,ed,c66a9980) at mi_switch+0x210
msleep(c6640e04,0,153,c0519238,0) at msleep+0x432
nfs_rcvlock(c9ba8800,0,0,c6640e00,0) at nfs_rcvlock+0x7d
nfs_reply(c9ba8800,c615bc40,c20fe100,c9ba8800,1c) at nfs_reply+0x2b
nfs_request(c9ba1c8c,c20fc000,7,c66a9980,c667a380) at nfs_request+0x2dc
nfs_writerpc(c9ba1c8c,e6daba2c,c667a380,e6daba1c,e6daba18) at nfs_writerpc+0x259
nfs_doio(d286c360,c667a380,c66a9980,0,c9ba1c8c) at nfs_doio+0x58e
nfs_strategy(e6dabaa0,1,c0511368,35e,16c) at nfs_strategy+0xd7
bwrite(d286c360,16c,0,2000,c66a9980) at bwrite+0x3af
nfs_write(e6dabbc8,c04f43a8,c057e3a0,c9ba1c8c,e6dabc7c) at nfs_write+0x86b
VOP_WRITE(c9ba1c8c,e6dabc7c,1,c667a380,0) at VOP_WRITE+0x5b
vn_write(c668bb04,e6dabc7c,c667a380,0,c66a9980) at vn_write+0x178
dofilewrite(c66a9980,c668bb04,7,80e1a00,14) at dofilewrite+0xe8
write(c66a9980,e6dabd10,c0526328,3fb,3) at write+0x69
syscall(2f,2f,2f,80d9b00,80d9b00) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (4, FreeBSD ELF32, write), eip = 0x80a80df, esp = 0xbfbff12c, ebp = 0xbfbff148 ---
db> panic
panic: from debugger
syncing disks, buffers remaining... VOP_LOCK: 0xc66846d8 is not locked but should be
VOP_GETVOBJECT: 0xc66846d8 is not locked but should be
VOP_GETVOBJECT: 0xc66846d8 is not locked but should be
VOP_UNLOCK: 0xc66846d8 is not locked but should be
VOP_LOCK: 0xc662536c is not locked but should be
VOP_FSYNC: 0xc662536c is not locked but should be
VOP_FSYNC: 0xc662536c is not locked but should be
VOP_UNLOCK: 0xc662536c is not locked but should be
VOP_LOCK: 0xc662536c is not locked but should be
VOP_INACTIVE: 0xc662536c is not locked but should be
VOP_UNLOCK: 0xc662536c is not locked but should be
VOP_LOCK: 0xc6632124 is not locked but should be
VOP_FSYNC: 0xc6632124 is not locked but should be
VOP_FSYNC: 0xc6632124 is not locked but should be
VOP_UNLOCK: 0xc6632124 is not locked but should be
VOP_LOCK: 0xc6645000 is not locked but should be
VOP_FSYNC: 0xc6645000 is not locked but should be
VOP_FSYNC: 0xc6645000 is not locked but should be
VOP_UNLOCK: 0xc6645000 is not locked but should be
VOP_LOCK: 0xc66af248 is not locked but should be
VOP_FSYNC: 0xc66af248 is not locked but should be
VOP_FSYNC: 0xc66af248 is not locked but should be
VOP_UNLOCK: 0xc66af248 is not locked but should be
VOP_LOCK: 0xc66bdc8c is not locked but should be
VOP_FSYNC: 0xc66bdc8c is not locked but should be
VOP_FSYNC: 0xc66bdc8c is not locked but should be
VOP_UNLOCK: 0xc66bdc8c is not locked but should be
VOP_LOCK: 0xc66bdc8c is not locked but should be
VOP_INACTIVE: 0xc66bdc8c is not locked but should be
VOP_UNLOCK: 0xc66bdc8c is not locked but should be
VOP_LOCK: 0xc681f248 is not locked but should be
VOP_FSYNC: 0xc681f248 is not locked but should be
VOP_FSYNC: 0xc681f248 is not locked but should be
VOP_UNLOCK: 0xc681f248 is not locked but should be
VOP_LOCK: 0xc681f248 is not locked but should be
VOP_INACTIVE: 0xc681f248 is not locked but should be
VOP_UNLOCK: 0xc681f248 is not locked but should be
panic: KSE not on run queue
Uptime: 1h23m55s
Dumping 1023 MB
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008
Dump complete
Terminate ACPI
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
More information about the freebsd-current
mailing list