FreeBSD 8.1-Release NFSD hang in rc_lo state

[alan bryan]

I'm doing some testing with loading up a new storage server.  I have couple ZFS filesystems exported via NFS over UDP.  

I have a client machine (8.1 also) that has mounted those filesystems along with some test PHP scripts that are doing a ton of read/write/fstat operations to load it up.

If I ctrl-C to kill the scripts on the client I found that I can end up with nfsd on the server stuck at 100% CPU and in the rc_lo state.  /etc/rc.d/nfsd restart does nothing.

# nfsstat -s -w 1 -W
 GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir
      0      0      0      0      0      0      0      0
      0      0      0      0      0      0      0      0
      0      0      0      0      0      0      0      0


Client complaining with:
kernel: nfs server 192.168.1.2:/tank/alantest2: not responding



Top on the server:

last pid:  4776;  load averages:  1.24,  1.15,  1.15                          up 0+23:04:46  18:25:26
53 processes:  2 running, 35 sleeping, 16 lock
CPU:  0.0% user,  0.0% nice, 25.0% system,  0.0% interrupt, 75.0% idle
Mem: 21M Active, 5812K Inact, 2029M Wired, 88K Cache, 8592K Buf, 5857M Free
Swap: 3942M Total, 3942M Free

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
  922 root      44    0  5804K  1820K CPU2    2  63:52 100.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  1  30:23  0.00% {nfsd: service}
  922 root      48    0  5804K  1820K rpcsvc  1  30:20  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  1  30:17  0.00% {nfsd: service}
  922 root      48    0  5804K  1820K rpcsvc  1  30:17  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K rpcsvc  1  30:13  0.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  2  30:13  0.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  3  30:12  0.00% {nfsd: service}
  922 root      44    0  5804K  1820K *rc_lo  0  30:10  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  2  30:07  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  0  30:03  0.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  1  30:03  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  2  30:03  0.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  2  30:02  0.00% {nfsd: service}
  922 root      45    0  5804K  1820K *rc_lo  1  29:59  0.00% {nfsd: master}
  922 root      45    0  5804K  1820K *rc_lo  2  29:54  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  0  29:54  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  0  29:52  0.00% {nfsd: service}
  922 root      47    0  5804K  1820K *rc_lo  2  29:46  0.00% {nfsd: service}
  922 root      46    0  5804K  1820K *rc_lo  1  29:44  0.00% {nfsd: service}