Odd network issues on ZFS based NFS server

[Rick Macklem]

On Tue, 8 Jun 2010, Anders Nordby wrote:

> Hi!
>
> I have a file server running 8.1-PRERELEASE amd64, where I share some
> filesystems using NFS and Samba. After running for a day or two, the
> server starts to get around 25% packet loss, browsing directories across
> NFS gets really slow etc. Rebooting solves it until it happens again.
> Has anyone experienced anything similar? I had this issue in FreeBSD 7
> as well, upgrading did not help.
>
Well, here's a few things you might try. (I know nothing about ZFS,
except what I see discussed on the mailing lists.)

- "netstat -m" will show you mbuf allocations. Might give you a hint
   w.r.t. mbuf/mbuf cluster exhaustion.
- I'd try setting zio_use_uma = 0, since there have been reports of
   issues related to ZFS using the uma allocator and mbuf allocation
   uses the uma allocator now, too. (I think this is fairly recent, so
   might not be relevant to FreeBSD7.)
- You can try the experimental NFS server to see if that affects the
   behaviour. ("-e" option on both mountd and nfsd)
- If you have some different network hardware, you could try a different
   net interface. This would isolate the problem, if it happens to be
   related to the network device driver for the hardware you have.

There are lots of email messages in the archive related to tuning the
arc for zfs. I know nothing about it, but I'd look for a message that
describes what the current recommendations are for amd64 w.r.t. this.

Hopefully others can suggest other things to check. It smells like some
sort of resource exhaustion problem, but who knows???

Good luck with it, rick