newnfsd's stuck on "ufs"

[Don Lewis]

On  5 Mar, rick at snowhite.cis.uoguelph.ca wrote:
> I'm working away at porting my NFSv4 server to FreeBSD5.2. It goes along
> ok for a while, but when doing several ops concurrently, most of the nfsd
> threads end up stuck sleeping on "ufs" as shown by the attached "ps axl".
> (The amusing part is that, once all but one thread is stuck, the last
>  thread works fine. In other words, the "fix" is to only run one newnfsd:-)
> 
> Anybody happen to know off the top of head, what I've screwed up?
> 
> Thanks for any hints, rick
> --- ps axl of newnfsd ---
>     0   523   522   0   4  0  1192  736 nfsd   I     ??    0:05.34 newnfsd: ser
>     0   524   522   0  -4  0  1192  736 ufs    D     ??    0:00.02 newnfsd: ser
>     0   525   522   0  -4  0  1192  736 ufs    D     ??    0:00.02 newnfsd: ser

Most likely a vnode locking bug.  Rebuild the kernel with
	options         DEBUG_VFS_LOCKS
	options         DDB

and see what it turns up.

There are some tuneable variables you can tweak in DDB to tune the
behaviour of the debugging code.

int vfs_badlock_ddb = 1;        /* Drop into debugger on violation. */
int vfs_badlock_mutex = 1;      /* Check for interlock across VOPs. */
int vfs_badlock_print = 1;      /* Print lock violations. */

The default is to drop into DDB every time a vnode lock assertion is
violated.  If you turn off vfs_badlock_ddb, the assertion violations
will just spam the console.