nve related LOR triggered by lots of small packets, and a hard hang

[Mark Atkinson]

Pyun YongHyeon wrote:
> On Fri, Feb 09, 2007 at 09:23:41AM -0800, Mark Atkinson wrote:
>  > Mark Atkinson wrote:
>  > 
>  > > Pyun YongHyeon wrote:
>  > > 
>  > >> On Wed, Jan 10, 2007 at 06:53:31PM +0300, Sergey Zaharchenko wrote:
>  > >>  > Hello John!
>  > >>  > 
>  > >>  > Wed, Jan 10, 2007 at 09:10:12AM -0500 you wrote:
>  > >>  > [snip]
>  > >>  > > Have you tried using nfe(4)? :)
>  > >>  > 
>  > >>  > Now I have, and it works just fine, thanks (I somehow thought nfe
>  > >>  > was specific to some platform). Why isn't it the default? Smaller
>  > >>  > range of hardware supported?
>  > >>  > 
>  > >> 
>  > >> AFAIK, nfe(4) supports more hardwares than that of nve(4).
>  > >> Try overhauled nfe(4) in the following URL.
>  > >> 
>  > >> http://people.freebsd.org/~yongari/nfe/if_nfe.c
>  > >> http://people.freebsd.org/~yongari/nfe/if_nfereg.h
>  > >> http://people.freebsd.org/~yongari/nfe/if_nfevar.h
>  > >> 
>  > >> The patch fixed serveral bugs in nfe(4) and it should perform better
>  > >> than nve(4). The following hardware features are supported.
>  > >>  o TSO
>  > >>  o Tx/Rx IP/TCP/UDP checksum offload
>  > >>  o VLAN hardware tag insertion/stripping
>  > >>  o Jumbo frame(up to 9100 bytes)
>  > >> 
>  > >> It seems that the hardware supports MSI/MSI-X too but I don't have
>  > >> nForce hardwares that supports MSI/MSI-X so it's hard to implement/
>  > >> experiment it. Accoring to the Shigeaki Tagashira, the author of
>  > >> FreeBSD nfe(4), his hardware claims to support 8 messages. I've
>  > >> checked Linux forcedeth driver to get hardware information for
>  > >> MSI/MSI-X but it I cound't understand the details. :-(
>  > >> 
>  > > 
>  > > I've been running into this hardlock LOR a lot recently on a  TYAN
>  > > 2895
>  > > (K8WE) based box.   So I tried your patch to nfe on today's -current.
>  > >   I tried a couple of small packet ping floods to a lan neighbor
>  > > under nfe and
>  > > it survived.   Did fine with some large NFS over TCP transfers as
>  > > well. However, I'll leave it up and running to see if it keels over
>  > > in the future.

[nfe dmesg snipped for brevity]

>  > After a day of running this, it became obvious the nfe driver patch has
>  > some
>  > sort of issue, at least with -current and this board.  Although NFS
>  > speeds seemed reasonable, transfers over TCP from a webserver suffered
>  > some sort
>  > of very noticeable pause/send/pause/send...  type problem that reduced
>  > transfers to about 6Kbyte/s.  This problem went away when putting nve
>  > back into the kernel and retrying the same scenerio.
>  > 
> 
> Would you explain the scenario to reproduce it on my box?


It didn't seem to be any sort of TCP induced delay (window sizes on both
ends of the connection were suitable throughout the transfer), seemed more
like an interrupt throttling or driver processing delay.   

With nve a large file transfer through nve0 over http with it acting as the
server would show

interrupt                          total       rate
irq21: nve0 ehci0                1254194          42

however during the same transfer with nfe I'd get only a rate of 5

Here are my kernel differences.   The difference between the two is simply
commenting out nve and enabling nfe or vice versa.

I'm running natd with divert -- no dummynet or other ipfw functionality is
enabled (it's there only for occasional traffic tests on this machine). I
should possibly retry without divert/ipfw over nfe0 as well.   vlan is also
in kernel but not used.

[root at marka-k8we conf]$ diff -u GENERIC K8WE
--- GENERIC     Wed Feb  7 12:14:17 2007
+++ K8WE        Fri Feb  9 08:56:18 2007
@@ -18,10 +18,10 @@
 #
 # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.461 2007/02/07 18:55:30 marcel
Exp $

-cpu            I486_CPU
-cpu            I586_CPU
+#cpu           I486_CPU
+#cpu           I586_CPU
 cpu            I686_CPU
-ident          GENERIC
+ident          K8WE

 # To statically compile in device wiring instead of /boot/device.hints
 #hints         "GENERIC.hints"         # Default places to look for
devices.
@@ -295,3 +295,12 @@
 device         firewire        # FireWire bus code
 device         sbp             # SCSI over FireWire (Requires scbus and da)
 device         fwe             # Ethernet over FireWire (non-standard!)
+
+#K8WE options
+options        IPFIREWALL
+options        IPDIVERT
+options        DUMMYNET
+options        IPFIREWALL_DEFAULT_TO_ACCEPT
+
+device         vlan
+device         nfe

-- 
Mark Atkinson
atkin901 at yahoo.com
(!wired)?(coffee++):(wired);