NFS trouble on 7.3-STABLE i386

[Jeremy Chadwick]

On Fri, May 21, 2010 at 07:45:47AM -0700, Mark Morley wrote:
> Having an issue with a file server here (7.3-STABLE i386)
> 
> The nfsd processes are hanging.  Client access to the nfs shares stops
> working and the nfsd processes on the server cannot be killed by any
> means.  There are no errors showing up anywhere on the server.  The
> network connection to the server seems fine (ie: anything other than
> nfs traffic seems ok).  Rebooting the server fixes the problem for a
> while, but it doesn't reboot easily.  It times out on terminating the
> nfsd processes.  When it finally does reboot the file system isn't
> marked clean, resulting in a long wait for fsck (although it doesn't
> find any problems, it's a multi terrabyte share and it takes a while).

I can't explain the dirty filesystem problem, especially if the server
does reboot/shut down properly.

> This morning it did it again.  This time I tried manually killing nfsd
> but nothing I did would make them die.  No errors.
>
> ... 
> 
> Any thoughts?

1) Are you forcing TCP or UDP NFS, or just using the default?

2) Is RPC still working?  Try running rpcinfo on both the client and
server.

3) Using rpcinfo and netstat, figure out what TCP or UDP port the client
is communicating with on the server, then use tcpdump to sniff traffic
on both the client and server specific to those port numbers and see if
there's any network I/O happening.

4) On the server, ktrace -t + -p {nfsd-pid} (I'm not sure which of the
two (master vs.  server) though) to see if anything is going on.

Rick Macklem probably has some better ideas than these though.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |