kern/177335: Sleeping on "vmopar" with the following non-sleepable locks held: exclusive sleep mutex NFSnode lock (NFSnode lock)

Sun Mar 24 11:30:01 UTC 2013

>Number:         177335
>Category:       kern
>Synopsis:       Sleeping on "vmopar" with the following non-sleepable locks held: exclusive sleep mutex NFSnode lock (NFSnode lock)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 24 11:30:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Svetlin Ivanov
>Release:        FreeBSD 8.3-STABLE
>Organization:
Global Electronic Solutions LTD.
>Environment:
FreeBSD www-02.gesbg.com 8.3-STABLE FreeBSD 8.3-STABLE #0: Tue Mar 12 12:40:17 EET 2013     root at GainesTown.gesbg.com:/usr/obj/usr/src/sys/GESBG80  amd64

>Description:
Hi list,

 We have 5 freebsd 8.3-STABLE (cvsuped Mar 12) that keeps crashing in a bad way.

 The are doing heavy apache/ nginx/ php5 web serving from a nfs mount and panic at least once a day with the following message:

1st 0xffffff0174f10ca8 vm object (standard object) @ /usr/src/sys/vm/vnode_pager.c:375
 2nd 0xffffffff80796020 Giant (Giant) @ /usr/src/sys/modules/usb/ukbd/../../../dev/usb/input/ukbd.c:2017
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x80a
_mtx_lock_flags() at _mtx_lock_flags+0x85
ukbd_poll() at ukbd_poll+0x44
kbdmux_poll() at kbdmux_poll+0x3f
sc_cngetc() at sc_cngetc+0xed
cncheckc() at cncheckc+0x65
cngetc() at cngetc+0x1e
db_readline() at db_readline+0x77
db_read_line() at db_read_line+0x15
db_command_loop() at db_command_loop+0x38
db_trap() at db_trap+0x89
kdb_trap() at kdb_trap+0xe1
trap() at trap+0x188
calltrap() at calltrap+0x8
--- trap 0x3, rip = 0xffffffff802efa2b, rsp = 0xffffff8362f99040, rbp = 0xffffff8362f99060 ---
kdb_enter() at kdb_enter+0x3b
witness_warn() at witness_warn+0x2c4
_sleep() at _sleep+0x64
vm_object_page_remove() at vm_object_page_remove+0x1df
vnode_pager_setsize() at vnode_pager_setsize+0x11f
nfs_loadattrcache() at nfs_loadattrcache+0x4e6
nfsm_loadattr_xx() at nfsm_loadattr_xx+0x2b
nfs_getattr() at nfs_getattr+0x2e9
VOP_GETATTR_APV() at VOP_GETATTR_APV+0xaf
nfs_lookup() at nfs_lookup+0x43a
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb3
lookup() at lookup+0x3d5
namei() at namei+0x481
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
amd64_syscall() at amd64_syscall+0x2b7
Xfast_syscall() at Xfast_syscall+0xfc
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x801488f5c, rsp = 0x7fffffffa608, rbp = 0x7fffffffa700 ---

db> gdb
(ctrl-c will return control to ddb)
Switching to gdb back-end

These servers are diskless and we are using online remote console debug.

[root at GainesTown /usr/obj/usr/src/sys/GESBG80]# kgdb -r /dev/cuau0 ./kernel.debug 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...Switching to remote protocol
kdb_enter (why=0xffffffff805acb07 "witness", msg=0xa <Address 0xa out of bounds>) at /usr/src/sys/kern/subr_kdb.c:454
454                     kdb_why = KDB_WHY_UNSET;

(kgdb) bt
#0  kdb_enter (why=0xffffffff805acb07 "witness", msg=0xa <Address 0xa out of bounds>) at /usr/src/sys/kern/subr_kdb.c:454
#1  0xffffffff80304d14 in witness_warn (flags=<value optimized out>, lock=0xffffff0174f10ca8, fmt=0xffffffff805a6178 "Sleeping on \"%s\"") at /usr/src/sys/kern/subr_witness.c:1691
#2  0xffffffff802c55e4 in _sleep (ident=0xffffff031b8dc990, lock=0xffffff0174f10ca8, priority=68, wmesg=0xffffffff805dba6e "vmopar", timo=0) at /usr/src/sys/kern/kern_synch.c:163
#3  0xffffffff8050038f in vm_object_page_remove (object=0xffffff0174f10ca8, start=0, end=22, clean_only=0) at vm_page.h:359
#4  0xffffffff8050b84f in vnode_pager_setsize (vp=<value optimized out>, nsize=0) at /usr/src/sys/vm/vnode_pager.c:389
#5  0xffffffff804683a6 in nfs_loadattrcache (vpp=<value optimized out>, mdp=<value optimized out>, dposp=<value optimized out>, vaper=0xffffff8362f993b0, dontshrink=0)
    at /usr/src/sys/nfsclient/nfs_subs.c:609
#6  0xffffffff804684bb in nfsm_loadattr_xx (v=0xffffff8362f99498, va=<value optimized out>, md=<value optimized out>, dpos=<value optimized out>)
    at /usr/src/sys/nfsclient/nfs_subs.c:973
#7  0xffffffff80470959 in nfs_getattr (ap=<value optimized out>) at /usr/src/sys/nfsclient/nfs_vnops.c:726
#8  0xffffffff8056d9bf in VOP_GETATTR_APV (vop=0xffffffff8074a720, a=0xffffff8362f996b0) at vnode_if.c:699
#9  0xffffffff8047442a in nfs_lookup (ap=<value optimized out>) at vnode_if.h:309
#10 0xffffffff8056e073 in VOP_LOOKUP_APV (vop=0xffffffff8074a720, a=0xffffff8362f997e0) at vnode_if.c:122
#11 0xffffffff8034ac95 in lookup (ndp=0xffffff8362f99900) at vnode_if.h:54
#12 0xffffffff8034bc71 in namei (ndp=0xffffff8362f99900) at /usr/src/sys/kern/vfs_lookup.c:269
#13 0xffffffff8035b18f in kern_statat_vnhook (td=0xffffff000adbd470, flag=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>, 
    sbp=0xffffff8362f99a80, hook=0) at /usr/src/sys/kern/vfs_syscalls.c:2339
#14 0xffffffff8035b335 in kern_statat (td=<value optimized out>, flag=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>, 
    sbp=<value optimized out>) at /usr/src/sys/kern/vfs_syscalls.c:2320
#15 0xffffffff8035b3fa in lstat (td=<value optimized out>, uap=0xffffff8362f99bb0) at /usr/src/sys/kern/vfs_syscalls.c:2383
#16 0xffffffff805300e7 in amd64_syscall (td=0xffffff000adbd470, traced=0) at subr_syscall.c:114
#17 0xffffffff8051750c in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387
#18 0x0000000801488f5c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

root at www-02:~ # mount
/dev/md0 on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
/dev/md1 on /etc (ufs, local)
tmpfs on /tmp (tmpfs, local, noexec)
/dev/md2 on /usr (ufs, local, soft-updates)
/dev/md3 on /var (ufs, local)
10.10.10.1:/storage/jails/store-01/usr/lib on /usr/lib (nfs)
10.10.10.1:/storage/jails/store-01/usr/local on /usr/local (nfs)
10.10.10.1:/storage/jails/store-01/storage on /storage (nfs, noatime)

>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: