FreeBSD 8.1-Release NFSD hang in rc_lo state

[alan bryan]

--- On Tue, 8/10/10, Rick Macklem <rmacklem at uoguelph.ca> wrote:

> From: Rick Macklem <rmacklem at uoguelph.ca>
> Subject: Re: FreeBSD 8.1-Release NFSD hang in rc_lo state
> To: "alan bryan" <alan.bryan at yahoo.com>
> Cc: freebsd-stable at freebsd.org
> Date: Tuesday, August 10, 2010, 7:35 PM
> > I'm doing some testing with
> loading up a new storage server. I have
> > couple ZFS filesystems exported via NFS over UDP.
> > 
> > I have a client machine (8.1 also) that has mounted
> those filesystems
> > along with some test PHP scripts that are doing a ton
> of
> > read/write/fstat operations to load it up.
> > 
> > If I ctrl-C to kill the scripts on the client I found
> that I can end
> > up with nfsd on the server stuck at 100% CPU and in
> the rc_lo state.
> > /etc/rc.d/nfsd restart does nothing.
> > 
> > # nfsstat -s -w 1 -W
> > GtAttr Lookup Rdlink Read Write Rename Access Rddir
> > 0 0 0 0 0 0 0 0
> > 0 0 0 0 0 0 0 0
> > 0 0 0 0 0 0 0 0
> > 
> > 
> > Client complaining with:
> > kernel: nfs server 192.168.1.2:/tank/alantest2: not
> responding
> > 
> > 
> > 
> > Top on the server:
> > 
> > last pid: 4776; load averages: 1.24, 1.15, 1.15 up
> 0+23:04:46 18:25:26
> > 53 processes: 2 running, 35 sleeping, 16 lock
> > CPU: 0.0% user, 0.0% nice, 25.0% system, 0.0%
> interrupt, 75.0% idle
> > Mem: 21M Active, 5812K Inact, 2029M Wired, 88K Cache,
> 8592K Buf, 5857M
> > Free
> > Swap: 3942M Total, 3942M Free
> > 
> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> > 922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd:
> service}
> > 922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd:
> service}
> > 922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd:
> service}
> > 922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd:
> service}
> > 922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd:
> master}
> > 922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd:
> service}
> > 922 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd:
> service}
> > 922 root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd:
> service}
> > 
> 
> I have a patch that I think might fix this. (Essentially
> the same
> bug was posted on another list recently.) I'll post it to
> Alan
> separately, but if anyone else wants to try it, it will be
> at
>    http://people.freebsd.org/~rmacklem/freebsd8.1-patches/replay.patch
> in a few minutes.
> 
> rick
> 
> 
> _______________________________________________
> freebsd-stable at freebsd.org
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> 





I applied this patch and have been unable to reproduce the hang since.

Rick, Thanks for an awesomely quick response!

This will go into 8-Stable at some point?

--Alan