Odd network issues on ZFS based NFS server

[Jeremy Chadwick]

On Tue, Jun 08, 2010 at 10:36:49AM +0200, Anders Nordby wrote:
> I have a file server running 8.1-PRERELEASE amd64, where I share some
> filesystems using NFS and Samba. After running for a day or two, the
> server starts to get around 25% packet loss, browsing directories across
> NFS gets really slow etc. Rebooting solves it until it happens again.
> Has anyone experienced anything similar? I had this issue in FreeBSD 7
> as well, upgrading did not help.
> 
> PS: I used mountd and /etc/exports to share the filesystems. I also
> regularly run zpool status from monitoring systems. I also replaced the
> server physically, changed switch ports, cables etc. So it does not seem
> to be a problem with hardware.

For what it's worth, we have a similar setup, but without Samba.
Machine happens to be running 8.0-STABLE (world/kernel Mon Apr 26
02:26:36).  No packet loss seen, and no overall issues aside from some
input errors on our em1 NIC (which do not correlate on the switch its
connected to).  NFS is not used heavily aside from daily backup jobs
across a gigE nework.  Machine has been up 42 days.

$ netstat -ibn
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll
em0    1500 <Link#1>      XX:XX:XX:XX:XX:XX  1541235     0     0  344966704   359378     0  255337127     0
em0    1500 XX.XX.XX.XX/X XX.XX.XX.XX         424637     -     -  271854885   359033     -  250287803     -
em1    1500 <Link#2>      XX:XX:XX:XX:XX:XX 62851814    59     0 81941673228 43418668     0 3520821042     0
em1    1500 XX.XX.XX.XX/X XX.XX.XX.XX       62813816     -     - 81059999660 43464778     - 2912408432     -
lo0   16384 <Link#3>                            3288     0     0     435566     3288     0     435566     0
lo0   16384 127.0.0.0/8   127.0.0.1             3288     -     -     435566     3288     -     435566     -

$ netstat -m
3085/2165/5250 mbufs in use (current/cache/total)
2549/2433/4982/25600 mbuf clusters in use (current/cache/total/max)
2048/896 mbuf+clusters out of packet secondary zone in use (current/cache)
0/104/104/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
5869K/5823K/11692K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

switch# show interfaces 2

 Status and Counters - Port Counters for port 2

  Name  : XXXXXXXXXX
  Link Status     : Up
  Totals (Since boot or last clear) :
   Bytes Rx        : 2,512,873,738      Bytes Tx        : 1,117,402,842
   Unicast Rx      : 169,702,228        Unicast Tx      : 237,760,196
   Bcast/Mcast Rx  : 12,667             Bcast/Mcast Tx  : 75,726
  Errors (Since boot or last clear) :
   FCS Rx          : 0                  Drops Rx        : 0
   Alignment Rx    : 0                  Collisions Tx   : 0
   Runts Rx        : 0                  Late Colln Tx   : 0
   Giants Rx       : 0                  Excessive Colln : 0
   Total Rx Errors : 0                  Deferred Tx     : 0
  Rates (5 minute weighted average) :
   Total Rx  (bps) : 1449296            Total Tx  (bps) : 1504528
   Unicast Rx (Pkts/sec) : 0            Unicast Tx (Pkts/sec) : 0
   B/Mcast Rx (Pkts/sec) : 0            B/Mcast Tx (Pkts/sec) : 0
   Utilization Rx  : 00.04 %            Utilization Tx  : 00.04 %


-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |