1168 octets payload and bad TCP checksums

[Kelly Yancey]

  We've got Broadcom BCM5701 cards configured for vlan tagging on a
FreeBSD 4.7 based router; a vlan switch then terminates the trunked
segment and splits it into separate physical subnets.  It turns out that
hosts on those segments cannot receive TCP packets with precisely 1168
octets of payload (ethernet frame size 1222 octets) as the checksum is
always incorrect.  We've manually backported all of the bge driver updates
from 4-stable, but to no avail.
  What is particularly odd is that the checksums are always wrong by the
same amount: 0xAC48 (the dump below only shows retries of the same
packet, but the difference is the same even for other packets).
Furthermore, it appears only TCP packets with 1168 octets of data are
affected.  I cannot easily create an environment without the vlans to
determine whether or not tagging is related.  Note also, that the IP
checksum is correct.

  Has anyone else experienced similar problems?  Does anyone have a clue
where to begin to track down the problem?  Currently I'm looking at the
tcp checksum calculation (tcp_fillheaders), but I don't really see how
that could be the culprit as if such a bug existed there, it would affect
all interfaces and surely would have been noticed by now.  At the same
time, I don't see anywhere else offhand the problem could be.  Again, if
anyone has any advice, I would greatly appreciate it.  Thanks,

  Kelly


bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1504
        options=3<rxcsum,txcsum>
        inet 10.30.3.254 netmask 0xfffffff8 broadcast 10.30.3.255
        ether 00:00:5e:00:01:4b
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
vlan9: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 10.30.3.1 netmask 0xfffffff8 broadcast 10.30.3.7
        ether 00:00:5e:00:01:4b
        vlan: 9 parent interface: bge0
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 10.30.3.9 netmask 0xfffffff8 broadcast 10.30.3.15
        ether 00:00:5e:00:01:4b
        vlan: 10 parent interface: bge0

  Extract from tcpdump -vvv taken on host 216.69.90.56 connected to
FreeBSD router via vlan10 interface:

11:38:55.665425 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 561:2021(1460) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57881, len 1500)
11:38:55.666782 216.69.68.198.22 > 216.69.90.56.3335: P [tcp sum ok] 2021:2049(28) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57882, len 68)
11:38:55.666839 216.69.90.56.3335 > 216.69.68.198.22: . [tcp sum ok] 432:432(0) ack 2049 win 17520 (DF) (ttl 128, id 57057, len 40)
11:38:55.668899 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57883, len 1208)
11:38:55.920110 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57884, len 1208)
11:38:56.419788 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57885, len 1208)
11:38:56.442824 216.69.224.134 > 216.69.90.56: icmp: echo request (ttl 108, id 24195, len 92)
11:38:57.419622 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57886, len 1208)
11:38:58.098535 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok]  12575+ PTR? 56.90.69.216.in-addr.arpa. (43) (ttl 128, id 57060, len 71)
11:38:58.098868 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok]  12576+ PTR? 1.90.69.216.in-addr.arpa. (42) (ttl 128, id 57061, len 70)
11:38:58.102453 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok]  12575 NXDomain* q: PTR? 56.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (103) (ttl 59, id 43147, len 131)
11:38:58.103689 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok]  12576 NXDomain* q: PTR? 1.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (102) (ttl 59, id 63562, len 130)
11:38:59.419902 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57887, len 1208)
11:39:03.419776 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10]  (ttl 59, id 57888, len 1208)
11:39:06.305954 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 432:480(48) ack 2049 win 17520 (DF) (ttl 128, id 57062, len 88)
11:39:06.344820 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 480 win 14352 (DF) [tos 0x10]  (ttl 59, id 57889, len 40)
11:39:07.031807 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 480:528(48) ack 2049 win 17520 (DF) (ttl 128, id 57065, len 88)
11:39:07.035322 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 528 win 14352 (DF) [tos 0x10]  (ttl 59, id 57890, len 40)