TSO ethernet frame errors on 8-STABLE, was: bogus 0 len IP packet

[Attila Nagy]

Hi,

On 02/07/11 01:22, Pyun YongHyeon wrote:
> On Sun, Feb 06, 2011 at 11:54:49PM +0100, Ronald Klop wrote:
>> On Sat, 22 Jan 2011 00:01:47 +0100, Ronald Klop
>> <ronald-freebsd8 at klop.yi.org>  wrote:
>>
>>> On Tue, 18 Jan 2011 09:38:04 +0100,<sthaug at nethelp.no>  wrote:
>>>
>>>>>> So, does anyone have an idea why the IP length field would be set to
>>>>> 0
>>>>>> for these TCP/IP packets?
>>>>>>
>>>>>> Here's some info from Ronald w.r.t. his hardware. (All I can think
>>>>> of is
>>>>>> that he could try disabling TSO, etc?)
>>>>>>
>>>>>> Thanks in advance for any help with this, rick
>>>>>>
>>>>> It seems that issue came from TSO. Driver will set ip_len and
>>>>> ip_sum field to 0 before passing the TCP segment to controller.
>>>>> The failed length were 4446, 5858, 3034 and 4310 and the total
>>>>> number of such frames are more than 35k within 90 seconds. Since
>>>>> failed length 4310 is continuously repeated I guess there is edge
>>>>> case where em(4) didn't free failed TCP segment for TSO.
>>>>> I remember there was commit to HEAD(r217295) which could be related
>>>>> with this issue.
>>>> I'm seeing the same problem with Broadcom NetXtreme (bce) cards:
>>>>
>>>> bce0 at pci0:3:0:0:        class=0x020000 card=0x03421014 chip=0x164c14e4
>>>> rev=0x12 hdr=0x00
>>>>     vendor     = 'Broadcom Corporation'
>>>>     device     = 'Broadcom NetXtreme II Gigabit Ethernet Adapter
>>>> (BCM5708)'
>>>>     class      = network
>>>>     subclass   = ethernet
>>>>
>>>> This is with 8.2-PRERELEASE. Turning off TSO (ifconfig bce0 -tso)
>>>> removes the problem.
>>>>
>>>> Steinar Haug, Nethelp consulting, sthaug at nethelp.no
>>>> _______________________________________________
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>> I tried -tso and -txcsum in various combinations, but it didn't solve
>>> the problem. I wil look for another brand of network card to try. But
>>> this has to wait till monday when I'm at the office again.
>> I also used another network card (rl0) and it has the same problem with
>> NFS. I'm going to change some network cables to see if that helps. I have
>> some hints that there might be something wrong with that.
>>
> Hmm, given that rl(4) also shows the issue it seems the issue could
> be in TCP/IP stack, not in driver side. rl(4) is dumb device so
> network stack should do segmentation and checksum computation.
> I highly doubt the issue came from faulty cable since other users
> also reported the same issue.
> Unfortunately I have no clue yet and I was not able to reproduce it
> on my box. I vaguely guess some code in kernel changed the ip_len
> to 0 in the middle of transmission. Rick's captured traffic looks
> normal except 0 ip_len given that controller is computing checksum
> on the fly. If mbuf chain was corrupted(e.g. m_len == 0) driver
> would have failed to send those frames.
I can see something similar.
Attached are two pcaps, with TSO enabled/disabled (net.inet.tcp.tso).

NIC is:
bce0: <HP NC373i Multifunction Gigabit Server Adapter (B2)> mem 
0xfa000000-0xfbffffff irq 16 at device 0.0 on pci7
miibus0: <MII bus> on bce0
brgphy0: <BCM5708S 1000/2500BaseSX PHY> PHY 2 on miibus0
brgphy0:  1000baseSX-FDX, 2500baseSX-FDX, auto
bce0: Ethernet address: 00:1f:29:cc:97:62
bce0: [ITHREAD]

In a HP BL460c. The OS is a 8-STABLE/amd64, compiled two days ago.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: tso-disabled
Type: application/octet-stream
Size: 2549 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110523/0a2d43dc/tso-disabled.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tso-enabled
Type: application/octet-stream
Size: 2467 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110523/0a2d43dc/tso-enabled.obj