Unusual TCP/IP Packet Size

[Jeremy Chadwick]

On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote:
> 13.02.2013 17:25, Doug Hardie ??????????:
> > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the following interface:
> > 
> > msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > 	options=c011b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
> > 	ether 00:11:2f:2a:c7:03
> > 	inet 10.0.1.199 netmask 0xffffff00 broadcast 10.0.1.255
> > 	inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 
> > 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > 	media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
> > 	status: active
> > 
> > 
> > It sent the following packet:  (data content abbreviated)
> > 
> > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr 920110183], length 3946
> > 	0x0000:  4500 0f9e ea89 4000 4006 2a08 0a00 01c7  E..... at .@.*.....
> > 	0x0010:  0a00 0102 01bb ef4a ece1 680b ae37 1bbc  .......J..h..7..
> > 	0x0020:  8018 0410 3407 0000 0101 080a 17f3 8ff8  ....4...??????.
> > 
> > 
> > The indicated packet length is 3946 and the load of data shown is that size.  The MTU on both interfaces is 1500.  The receiving system received 3 packets.  There is a router and switch between them.  One of them fragmented that packet. This is part of a SSL/TLS exchange and one side or the other is hanging on this and just dropping the connection.  I suspect the packet size is the issue. ssldump complains about the packet too and stops monitoring.  Could this possibly be related to the hardware checksums?
> 
> You have TSO enabled on the interface, so large outgoing TCP packet is pretty normal.
> It will be split by the NIC. Disable TSO with ifconfig if it interferes with your ssldump.

This is not the behaviour I see with em(4) on a 82573E with all defaults
used (which includes TSO4).  Note that Doug is using msk(4).

I can provide packet captures on both ends of a LAN segment using both
tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that
show a difference in behaviour compared to what Doug sees.

What I see on the FreeBSD side with tcpdump is repeated "bad-len 0"
messages for payloads which are chunked or segmented as a result of TSO.
I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I
only see one "bad-len" entry for all chunks (up until the next ACK or
PSH+ACK of course).

The important part: I do not see captured TCP packets reporting a length
greater than MTU (or MSS for that matter (remember: IP header + TCP
header + MSS <= MTU).

Also note for Doug: remember that if you're doing packet captures
between two devices that have NAT involved, you may see different
behaviour.  Example case:

03:58:47.907582 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack 13419, win 64240, length 0
03:58:47.907649 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq 17799:19259, ack 292, win 1026, length 1460
03:58:47.907679 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq 19259:20719, ack 292, win 1026, length 1460
03:58:47.912546 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack 16339, win 64240, length 0

In the above example there's a Linux NAT router (67.180.84.87) with a
client (192.168.1.50) behind it, talking to 206.125.172.42.  MTU is 1500
(I obviously didn't include the initial SYN  :-) ).

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |