Zombie NFS writing from FreeBSD clients to FreeBSD 8.0 server with ZFS

[alan bryan]

I have a AMD64 FreeBSD 8.0 server with ZFS filesystem being shared via NFS.  These are being accessed by the clients.  The clients are a mix of FreeBSD 6.2 32bit and FreeBSD 7.0 64bit.  I have seen similar behavior from both versions of FreeBSD as clients.

The behavior that I'm seeing is that everything is fine for a period of time and then the client starts "writing" large amounts of data to the NFS server.  "Writing" is in quotes as nothing is actually being written to disk - this can go on for 12+ hrs or more - until the client is rebooted. It appears to cap out at around the 10-20Mbps rate and just sit there.  Other clients are fine during this time.

On the server:
# nfsstat -s -w 1
 GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir
      0     25      2     38   7661      0     48      0
      0     16      0      6   7601      0     14      0
      0     21      0     13   7541      0     19      0
(client apache is stopped - nfsstat on client shows no activity)
(server writes still continue)
      0     13      0      1   7331      0      5      0
      0     19      0     25   7479      0     59      0
      0      9      0      9   7777      0     23      0
      0     14      0     51   7640      0     33      0
      0      8      0     40   4476      0     25      0
(Right here is when the bad client is rebooted)
(everything is good again)
      4     26      1     66     21      0     31      0
      0      6      0     50     19      0      4      0
      0     15      0     86     23      0     32      0

On the clients (webservers) I killed apache and it appears that nothing is then writing any longer.  The same nfsstat -c -w 1 then shows zero activity.   However, the high write volume continues at the server until this broken client machine is rebooted - and then the large drop in writes as you see above.  At that point everything is now normal again.

Looking in /var/log/messages on the client and server in these time periods hasn't shown any errors.

Client mounts in /etc/fstab look like:
192.168.1.33:/tank/share /usr/local/www/share nfs rw,-b,-i,-U,-3 0 0

Any ideas on what to try, where to look for more insight, etc...??

Thanks,
Alan