directory listing hangs in "ufs" state

[Andrey Zonov]

Hi,

I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it.  A 
couple days ago I observed that listing of mongodb directory stuck in a 
few minutes in "ufs" state. I've run it again with ktrace and got 
following (kdump -R):

  91324 ls       0.000003 CALL  lstat(0x32c199c8,0x32c19950)
  91324 ls       0.000003 NAMI  "base.1"
  91324 ls       21.357255 STRU  struct stat {dev=116, ino=45125633, 
mode=-rw------- , nlink=1, uid=922, gid=922, rdev=180226648, 
atime=1323709877, stime=1323776461, ctime=1323776461, 
birthtime=1314798592, size=134217728, blksize=16384, blocks=262304, 
flags=0x0 }
  91324 ls       0.000014 RET   lstat 0

kgdb backtrace of this process was looked like this:

Thread 297 (Thread 100372):
#0  sched_switch (td=0xffffff0095c008c0, newtd=0xffffff000357b8c0, 
flags=) at /usr/src/sys/kern/sched_ule.c:1866
#1  0xffffffff80406696 in mi_switch (flags=260, newtd=0x0) at 
/usr/src/sys/kern/kern_synch.c:449
#2  0xffffffff8043c072 in sleepq_wait (wchan=0xffffff0103aaf7f8, pri=80) 
at /usr/src/sys/kern/subr_sleepqueue.c:609
#3  0xffffffff803e4a5a in __lockmgr_args (lk=0xffffff0103aaf7f8, 
flags=2097408, ilk=0xffffff0103aaf820, wmesg=) at 
/usr/src/sys/kern/kern_lock.c:220
#5  0xffffffff8061239c in ffs_lock (ap=0xffffff84867fc550) at lockmgr.h:94
#5  0xffffffff806d2462 in VOP_LOCK1_APV (vop=0xffffffff80921fe0, 
a=0xffffff84867fc550) at vnode_if.c:1988
#6  0xffffffff804a58b7 in _vn_lock (vp=0xffffff0103aaf760, 
flags=2097152, file=0xffffffff80736e70 "/usr/src/sys/kern/vfs_subr.c", 
line=2137) at vnode_if.h:859
#7  0xffffffff80498bc0 in vget (vp=0xffffff0103aaf760, flags=2097408, 
td=0xffffff0095c008c0) at /usr/src/sys/kern/vfs_subr.c:2137
#8  0xffffffff804845f4 in cache_lookup (dvp=0xffffff0095675b10, 
vpp=0xffffff84867fc910, cnp=0xffffff84867fc938) at 
/usr/src/sys/kern/vfs_cache.c:587
#9  0xffffffff80484a30 in vfs_cache_lookup (ap=) at 
/usr/src/sys/kern/vfs_cache.c:905
#10 0xffffffff806d2e7c in VOP_LOOKUP_APV (vop=0xffffffff80922820, 
a=0xffffff84867fc790) at vnode_if.c:123
#11 0xffffffff8048bc80 in lookup (ndp=0xffffff84867fc8e0) at vnode_if.h:54
#12 0xffffffff8048cf0e in namei (ndp=0xffffff84867fc8e0) at 
/usr/src/sys/kern/vfs_lookup.c:269
#13 0xffffffff8049c972 in kern_statat_vnhook (td=0xffffff0095c008c0, 
flag=) at /usr/src/sys/kern/vfs_syscalls.c:2346
#14 0xffffffff8049cbb5 in kern_statat (td=) at 
/usr/src/sys/kern/vfs_syscalls.c:2327
#15 0xffffffff8049cc7a in lstat (td=) at 
/usr/src/sys/kern/vfs_syscalls.c:2390
#16 0xffffffff8043e7dd in syscallenter (td=0xffffff0095c008c0, 
sa=0xffffff84867fcbb0) at /usr/src/sys/kern/subr_trap.c:326
#17 0xffffffff8066a5eb in syscall (frame=0xffffff84867fcc50) at 
/usr/src/sys/amd64/amd64/trap.c:916
#18 0xffffffff806517f2 in Xfast_syscall () at 
/usr/src/sys/amd64/amd64/exception.S:384
#19 0x000000003298f75c in ?? ()

The very first idea was to turn off name caching (set debug.vfscache to 
0), but it didn't help. The second idea was to reboot, but it didn't 
help too.

This directory locks fine. It has 10 files and 1 empty directory.

Have you any ideas what is going on? or how to catch the problem?

-- 
Andrey Zonov