Packet corruption in re0 [checksum offloading]

[Pyun YongHyeon]

On Sat, Feb 23, 2008 at 10:35:57AM -0800, Sam Leffler wrote:
 > Pyun YongHyeon wrote:
 > >On Fri, Feb 22, 2008 at 10:43:22AM +0200, Ian FREISLICH wrote:
 > > > Pyun YongHyeon wrote:
 > > > > On Thu, Feb 21, 2008 at 01:18:18PM +0200, Ian FREISLICH wrote:
 > > > >  > Pyun YongHyeon wrote:
 > > > >  > > On Thu, Feb 21, 2008 at 02:47:43PM +1000, Robert Backhaus wrote:
 > > > >  > >  > On Thu, Feb 21, 2008 at 1:50 PM, Pyun YongHyeon 
 > > <pyunyh at gmail.com> wr
 > > > ote:
 > > > >  > >  > > On Thu, Feb 21, 2008 at 11:03:02AM +1000, Robert Backhaus 
 > > wrote:
 > > > >  > >  > >   > I am experiencing roughly 15% packet corruption on the 
 > > re inter
 > > > face 
 > > > >  > on
 > > > >  > >  > >   > my freebsd 7/amd64  box.
 > > > >  > >  > >   >
 > > > >  > >  > >   > FreeBSD gw.flexi.robbak.com 7.0-PRERELEASE FreeBSD 
 > > 7.0-PRERELEA
 > > > SE #8
 > > > >  > :
 > > > >  > >  > >   > Tue Feb  5 09:49:55 EST 2008
 > > > >  > >  > >   > root at gw.flexi.robbak.com:/usr/obj/usr/src/sys/GW  amd64
 > > > >  > >  > >   >
 > > > >  > >  > >   > Just to make troubleshooting difficult, this problem 
 > > only shows
 > > >  up
 > > > >  > >  > >   > after the system has been up for roughly 36 hours, 
 > > depending on
 > > >  the
 > > > >  > >  > >   > amount of traffic.
 > > > >  > >  > >   >
 > > > >  > >  > >
 > > > >  > >  > >  I didn't take a look attached tcpdump files but I guess the
 > > > >  > >  > >  instability issue was fixed in HEAD. It's not yet MFCed but
 > > > >  > >  > >  I'll handle it in a week.
 > > > >  > >  > >
 > > > >  > >  > >  Would you try re(4) in HEAD?
 > > > >  > >  > >
 > > > >  > >  > 
 > > > >  > >  > OK, I'll do that. What is the best way to do that? csupping 
 > > to "." se
 > > > ems a
 > > > >  > >  > bit drastic, and I don't do much with cvs proper. I take it 
 > > that I sh
 > > > ould 
 > > > >  > use
 > > > >  > >  > anon-cvs to grab the directory, but I don't quite know how.
 > > > >  > >  > 
 > > > >  > > 
 > > > >  > > Copy sys/dev/re/if_re.c, sys/pci/if_rlreg.h in HEAD to your box.
 > > > >  > > Due to lack of m_defrag(9) in 7-PRERELEASE/RC, you also have to 
 > > add
 > > > >  > > that function to if_re.c(Copy m_defrag() in sys/kern/uipc_mbuf.c 
 > > on
 > > > >  > > HEAD/RELENG_7 to if_re.c). That would make it build on your box.
 > > > >  > 
 > > > >  > This doesn't solve the problem that I'm seeing on re(4) interfaces.
 > > > >  > It basically shows up as quagga establishing OSPF neighours as
 > > > >  > "Exchange/DR" when VLAN hardware tagging is enabled.  I'm running
 > > > >  > OSPF over 802.1Q vlans.  Neighbours are correctly negotiated once
 > > > >  > VLAN hardware tagging is disabled on the interface.
 > > > >  > 
 > > > >  > I'll do more debugging.
 > > > >  > 
 > > > > 
 > > > > Hmm. That sounds like different issue to me. I guess I din't change
 > > > > any semantics in VLAN H/W tagging. Do you still the same VLAN H/W
 > > > > tagging related issues on RELENG_7?
 > > > > 
 > > > > To narrow down the issue it would be even better to know which parts
 > > > > of H/W assistance was broken. For example,
 > > > >  - Disable checksum offload for VLAN interface first and check
 > > > >    whether quagga works.
 > > > 
 > > > You can only disable offload on the parent interface.
 > > > 
 > >
 > >Hmm... I thought it should work.
 > >I have no idea why ioctl handler of vlan(4) rejects checksum
 > >offload configutation. I guess vlan(4) should be teached to handle
 > >this. If parent interface have IFCAP_VLAN_HWCSUM capability and
 > >IFCAP_VLAN_HWTAGGING, ifconfig(4) should be able to control checksum
 > >offload for vlan(4) interface. CCed to yar to get his opinions on
 > >controlling checksum offload on vlan(4).
 > >
 > > > >  - Disable checksum offload for parent interface and check again.
 > > > > If you can post tcpdump output for broken conntection it may help a
 > > > > lot to diagnose the issue.
 > > > 
 > > > The only flag affecting this behaviour is vlanhwtag.  Various
 > > > permutations of the interface flags make no difference to this
 > > > behaviour as long as hardware tagging is enabled.
 > > > 
 > >
 > >Disabling VLAN HW tagging also turns off checksum offload on vlan(4)
 > >interface.
 > >
 > >  
 > 
 > This reminds me that there are several places in the system where h/w 
 > checksum offload needs to be specially handled but instead is disabled 
 > as a WAR.  In particular I'm thinking of the bridge where txcsum is 
 > muted on devices while they are plumbed.  But this can be a big loss and 
 > the better approach (IMO) is to fill in the missing capability in s/w.
 > 

Agreed.

 > Not sure what components there are besides bridge and vlan; maybe lagg?  
 > netgraph?
 > 

I'm not familiar with lagg(4) and netgraph(4). But lagg(4) should
disable Tx checksum offload if one of interface is not capable of
hardware checksum offload.

 > Note there are other capabilities besides checksum offload, TSO can be 
 > done in s/w with good effect.
 > 

AFAIK bridge(4) blindly disables Tx checksum offload for all
members of a bridge. If all members of a bridge can do checksum
offload/TSO with hardware assistance I guess there is no reason to
disable these capabilities in bridge environments. The same apply
to lagg(4) too.
S/W checksum offload/TSO emulation for intefaces without these
hardware capabilities would greatly enhance Tx performance when
other member of interface of a bridage can make use of hardware
offload capability.

 >    Sam
 > 

-- 
Regards,
Pyun YongHyeon