Packet corruption in re0 [checksum offloading]

[Sam Leffler]

Pyun YongHyeon wrote:
> On Sat, Feb 23, 2008 at 10:35:57AM -0800, Sam Leffler wrote:
>  > Pyun YongHyeon wrote:
>  > >On Fri, Feb 22, 2008 at 10:43:22AM +0200, Ian FREISLICH wrote:
>  > > > Pyun YongHyeon wrote:
>  > > > > On Thu, Feb 21, 2008 at 01:18:18PM +0200, Ian FREISLICH wrote:
>  > > > >  > Pyun YongHyeon wrote:
>  > > > >  > > On Thu, Feb 21, 2008 at 02:47:43PM +1000, Robert Backhaus wrote:
>  > > > >  > >  > On Thu, Feb 21, 2008 at 1:50 PM, Pyun YongHyeon 
>  > > <pyunyh at gmail.com> wr
>  > > > ote:
>  > > > >  > >  > > On Thu, Feb 21, 2008 at 11:03:02AM +1000, Robert Backhaus 
>  > > wrote:
>  > > > >  > >  > >   > I am experiencing roughly 15% packet corruption on the 
>  > > re inter
>  > > > face 
>  > > > >  > on
>  > > > >  > >  > >   > my freebsd 7/amd64  box.
>  > > > >  > >  > >   >
>  > > > >  > >  > >   > FreeBSD gw.flexi.robbak.com 7.0-PRERELEASE FreeBSD 
>  > > 7.0-PRERELEA
>  > > > SE #8
>  > > > >  > :
>  > > > >  > >  > >   > Tue Feb  5 09:49:55 EST 2008
>  > > > >  > >  > >   > root at gw.flexi.robbak.com:/usr/obj/usr/src/sys/GW  amd64
>  > > > >  > >  > >   >
>  > > > >  > >  > >   > Just to make troubleshooting difficult, this problem 
>  > > only shows
>  > > >  up
>  > > > >  > >  > >   > after the system has been up for roughly 36 hours, 
>  > > depending on
>  > > >  the
>  > > > >  > >  > >   > amount of traffic.
>  > > > >  > >  > >   >
>  > > > >  > >  > >
>  > > > >  > >  > >  I didn't take a look attached tcpdump files but I guess the
>  > > > >  > >  > >  instability issue was fixed in HEAD. It's not yet MFCed but
>  > > > >  > >  > >  I'll handle it in a week.
>  > > > >  > >  > >
>  > > > >  > >  > >  Would you try re(4) in HEAD?
>  > > > >  > >  > >
>  > > > >  > >  > 
>  > > > >  > >  > OK, I'll do that. What is the best way to do that? csupping 
>  > > to "." se
>  > > > ems a
>  > > > >  > >  > bit drastic, and I don't do much with cvs proper. I take it 
>  > > that I sh
>  > > > ould 
>  > > > >  > use
>  > > > >  > >  > anon-cvs to grab the directory, but I don't quite know how.
>  > > > >  > >  > 
>  > > > >  > > 
>  > > > >  > > Copy sys/dev/re/if_re.c, sys/pci/if_rlreg.h in HEAD to your box.
>  > > > >  > > Due to lack of m_defrag(9) in 7-PRERELEASE/RC, you also have to 
>  > > add
>  > > > >  > > that function to if_re.c(Copy m_defrag() in sys/kern/uipc_mbuf.c 
>  > > on
>  > > > >  > > HEAD/RELENG_7 to if_re.c). That would make it build on your box.
>  > > > >  > 
>  > > > >  > This doesn't solve the problem that I'm seeing on re(4) interfaces.
>  > > > >  > It basically shows up as quagga establishing OSPF neighours as
>  > > > >  > "Exchange/DR" when VLAN hardware tagging is enabled.  I'm running
>  > > > >  > OSPF over 802.1Q vlans.  Neighbours are correctly negotiated once
>  > > > >  > VLAN hardware tagging is disabled on the interface.
>  > > > >  > 
>  > > > >  > I'll do more debugging.
>  > > > >  > 
>  > > > > 
>  > > > > Hmm. That sounds like different issue to me. I guess I din't change
>  > > > > any semantics in VLAN H/W tagging. Do you still the same VLAN H/W
>  > > > > tagging related issues on RELENG_7?
>  > > > > 
>  > > > > To narrow down the issue it would be even better to know which parts
>  > > > > of H/W assistance was broken. For example,
>  > > > >  - Disable checksum offload for VLAN interface first and check
>  > > > >    whether quagga works.
>  > > > 
>  > > > You can only disable offload on the parent interface.
>  > > > 
>  > >
>  > >Hmm... I thought it should work.
>  > >I have no idea why ioctl handler of vlan(4) rejects checksum
>  > >offload configutation. I guess vlan(4) should be teached to handle
>  > >this. If parent interface have IFCAP_VLAN_HWCSUM capability and
>  > >IFCAP_VLAN_HWTAGGING, ifconfig(4) should be able to control checksum
>  > >offload for vlan(4) interface. CCed to yar to get his opinions on
>  > >controlling checksum offload on vlan(4).
>  > >
>  > > > >  - Disable checksum offload for parent interface and check again.
>  > > > > If you can post tcpdump output for broken conntection it may help a
>  > > > > lot to diagnose the issue.
>  > > > 
>  > > > The only flag affecting this behaviour is vlanhwtag.  Various
>  > > > permutations of the interface flags make no difference to this
>  > > > behaviour as long as hardware tagging is enabled.
>  > > > 
>  > >
>  > >Disabling VLAN HW tagging also turns off checksum offload on vlan(4)
>  > >interface.
>  > >
>  > >  
>  > 
>  > This reminds me that there are several places in the system where h/w 
>  > checksum offload needs to be specially handled but instead is disabled 
>  > as a WAR.  In particular I'm thinking of the bridge where txcsum is 
>  > muted on devices while they are plumbed.  But this can be a big loss and 
>  > the better approach (IMO) is to fill in the missing capability in s/w.
>  > 
> 
> Agreed.
> 
>  > Not sure what components there are besides bridge and vlan; maybe lagg?  
>  > netgraph?
>  > 
> 
> I'm not familiar with lagg(4) and netgraph(4). But lagg(4) should
> disable Tx checksum offload if one of interface is not capable of
> hardware checksum offload.

Nope, don't disable; provide s/w support for the interface.

> 
>  > Note there are other capabilities besides checksum offload, TSO can be 
>  > done in s/w with good effect.
>  > 
> 
> AFAIK bridge(4) blindly disables Tx checksum offload for all
> members of a bridge. If all members of a bridge can do checksum
> offload/TSO with hardware assistance I guess there is no reason to
> disable these capabilities in bridge environments. The same apply
> to lagg(4) too.
> S/W checksum offload/TSO emulation for intefaces without these
> hardware capabilities would greatly enhance Tx performance when
> other member of interface of a bridage can make use of hardware
> offload capability.

TSO over 802.11n networks is important even when done in software.  But 
the point here is that we're losing mucho performance by blindly 
disabling features instead of providing host replacements so members of 
the bridge (or other aggregate) operate w/ full functionality.

	Sam