Odd network issues on ZFS based NFS server

[Pyun YongHyeon]

On Sat, Jun 12, 2010 at 01:01:20AM +0200, Anders Nordby wrote:
> Hi,
> 
> On Fri, Jun 11, 2010 at 10:58:05AM -0700, Pyun YongHyeon wrote:
> >> Let me know if there's anything bge related I can try/test. It might
> >> take a day or two or more. Customer is sort of getting annoyed by these
> >> problems, so the room for testing is getting smaller. But of course I
> >> want to help get a fix for this.
> > Show me dmesg output to know which bge(4) controller you had. And
> > show me output of "netstat -ndI bge0". Some bge(4) controllers
> > supports detailed MAC counters and these are exported via sysctl.
> > If your controller is one of these controller, you can check the
> > statistics of controller with "sysctl dev.bge.0.stat" and post it
> > if you can see them.
> 
> Since running on rue NIC I didn't retry bge again. But I did not reboot
> since I had problems last time either, I just changed NIC from bge1 to
> ue0. So I'm not sure if these numbers are interesting or if I should
> retry using a bge NIC, but here goes:
> 
> anders at unixfile:~$ grep ^bge1 /var/run/dmesg.boot 
> bge1: <HP NC7782 Gigabit Server Adapter, ASIC rev. 0x002100> mem
> 0xfdce0000-0xfdceffff irq 26 at device 1.1 on pci3
> bge1: Ethernet address: 00:16:35:03:e6:3e
> bge1: [ITHREAD]
> anders at unixfile:~$ netstat -ndI bge1
> Name    Mtu Network       Address              Ipkts Ierrs Idrop
> Opkts Oerrs  Coll Drop
> bge1*  1500 <Link#2>      00:16:35:03:e6:3e 21417404     0     0
> 20313076     0     0    0 
> anders at unixfile:~$ sysctl dev.bge.1.stats
> dev.bge.1.stats.FramesDroppedDueToFilters: 0
> dev.bge.1.stats.DmaWriteQueueFull: 34
> dev.bge.1.stats.DmaWriteHighPriQueueFull: 0
> dev.bge.1.stats.NoMoreRxBDs: 0
> dev.bge.1.stats.InputDiscards: 0
> dev.bge.1.stats.InputErrors: 0
> dev.bge.1.stats.RecvThresholdHit: 12086131
> dev.bge.1.stats.DmaReadQueueFull: 957280
> dev.bge.1.stats.DmaReadHighPriQueueFull: 4835
> dev.bge.1.stats.SendDataCompQueueFull: 0
> dev.bge.1.stats.RingSetSendProdIndex: 20515417
> dev.bge.1.stats.RingStatusUpdate: 20492506
> dev.bge.1.stats.Interrupts: 20492506
> dev.bge.1.stats.AvoidedInterrupts: 0
> dev.bge.1.stats.SendThresholdHit: 0
> dev.bge.1.stats.rx.Octets: 0
> dev.bge.1.stats.rx.Fragments: 0
> dev.bge.1.stats.rx.UcastPkts: 0

This MAC statistics look odd. netstat(1) showed you had 21417404
input packets and 20313076 output packets. But these counters show
all 0s. I have no idea how this can happen. Did the controller ever
work? If the controller used to work before, I suspect hardware
failure(NIC part).

> dev.bge.1.stats.rx.MulticastPkts: 0
> dev.bge.1.stats.rx.FCSErrors: 0
> dev.bge.1.stats.rx.AlignmentErrors: 0
> dev.bge.1.stats.rx.xonPauseFramesReceived: 0
> dev.bge.1.stats.rx.xoffPauseFramesReceived: 0
> dev.bge.1.stats.rx.ControlFramesReceived: 0
> dev.bge.1.stats.rx.xoffStateEntered: 0
> dev.bge.1.stats.rx.FramesTooLong: 0
> dev.bge.1.stats.rx.Jabbers: 0
> dev.bge.1.stats.rx.UndersizePkts: 0
> dev.bge.1.stats.rx.inRangeLengthError: 0
> dev.bge.1.stats.rx.outRangeLengthError: 0
> dev.bge.1.stats.tx.Octets: 0
> dev.bge.1.stats.tx.Collisions: 0
> dev.bge.1.stats.tx.XonSent: 0
> dev.bge.1.stats.tx.XoffSent: 0
> dev.bge.1.stats.tx.flowControlDone: 0
> dev.bge.1.stats.tx.InternalMacTransmitErrors: 0
> dev.bge.1.stats.tx.SingleCollisionFrames: 0
> dev.bge.1.stats.tx.MultipleCollisionFrames: 0
> dev.bge.1.stats.tx.DeferredTransmissions: 0
> dev.bge.1.stats.tx.ExcessiveCollisions: 0
> dev.bge.1.stats.tx.LateCollisions: 0
> dev.bge.1.stats.tx.UcastPkts: 0
> dev.bge.1.stats.tx.MulticastPkts: 0
> dev.bge.1.stats.tx.BroadcastPkts: 0
> dev.bge.1.stats.tx.CarrierSenseErrors: 0
> dev.bge.1.stats.tx.Discards: 0
> dev.bge.1.stats.tx.Errors: 0
> 
> Regards,
> 
> -- 
> Anders.