Tcpdump says I'm getting incomplete packets; how to find the culprit?

[Dan Nelson]

In the last episode (Dec 28), Doug Lee said:
> On Mon, Dec 27, 2004 at 07:30:41PM -0600, Dan Nelson wrote:
> > In the last episode (Dec 27), Doug Lee said:
> > > I use FreeBSD 4.10-STABLE as a nat/firewall box.  When connected
> > > to DSL, I got fast web surfing but many gaps in incoming audio
> > > traffic using some audio software.  I switched to cable, and now
> > > audio works great, but at least when I pop open pages in Lynx
> > > right on the FreeBSD box, I often experience five-second
> > > delays--one at "202 OK" and one or more during the loading of the
> > > page.  Tcpdump reports that I'm receiving incomplete packets, so
> > > I assume the five-second delays are timeouts on my box before a
> > > request for packet resends.
> > 
> > What is tcpdump printing that makes you think that packets are
> > incomplete?  If you are manually decoding packets by looking at
> > tcpdump -X output, make sure you also use -s 0 to grab the entire
> > packet.
> 
> This is from a "tcpdump -s 0 -w tco -i ed0 port 80" run.  Line 12
> shows a truncation but no delay, interestingly enough; but I believe
> line 17 is the one that occurred when I saw "202 OK" and a
> five-second delay.  Actually, I guess it's a seven-second delay after
> all. :-) I replaced my ip with <me> here.

>  1 05:20:02.131687 <me>.4891 > 12.129.203.38.http: S 1518360911:1518360911(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 449963838 0> (DF)
>  2 05:20:02.211922 12.129.203.38.http > <me>.4891: S 1407738134:1407738134(0) ack 1518360912 win 10136 <nop,nop,timestamp 164687034 449963838,nop,wscale 0,mss 1460> (DF)
>  3 05:20:02.212540 <me>.4891 > 12.129.203.38.http: . ack 1 win 33304 <nop,nop,timestamp 449963919 164687034> (DF)
>  4 05:20:02.221406 <me>.4891 > 12.129.203.38.http: . 1:1449(1448) ack 1 win 33304 <nop,nop,timestamp 449963928 164687034> (DF)
>  5 05:20:02.311500 12.129.203.38.http > <me>.4891: . ack 1449 win 10136 <nop,nop,timestamp 164687044 449963928> (DF)
>  6 05:20:02.312173 <me>.4891 > 12.129.203.38.http: P 1449:1615(166) ack 1 win 33304 <nop,nop,timestamp 449964019 164687044> (DF)
>  7 05:20:02.397972 12.129.203.38.http > <me>.4891: . 1:1449(1448) ack 1615 win 10136 <nop,nop,timestamp 164687052 449964019> (DF)
>  8 05:20:02.399266 12.129.203.38.http > <me>.4891: P 1449:2897(1448) ack 1615 win 10136 <nop,nop,timestamp 164687052 449964019> (DF)
>  9 05:20:02.402194 <me>.4891 > 12.129.203.38.http: . ack 2897 win 32580 <nop,nop,timestamp 449964109 164687052> (DF)
> 10 05:20:02.485577 12.129.203.38.http > <me>.4891: . 2897:4345(1448) ack 1615 win 10136 <nop,nop,timestamp 164687061 449964109> (DF)
> 11 05:20:02.486357 <me>.4891 > 12.129.203.38.http: . ack 4345 win 33304 <nop,nop,timestamp 449964193 164687061> (DF)
> 12 05:20:02.486606 truncated-ip - 276 bytes missing! 12.129.203.38.http > <me>.4891: . 4345:5793(1448) ack 1615 win 10136 <nop,nop,timestamp 164687061 449964109> (DF)
> 13 05:20:02.487904 12.129.203.38.http > <me>.4891: P 5793:7241(1448) ack 1615 win 10136 <nop,nop,timestamp 164687061 449964109> (DF)
> 14 05:20:02.491372 <me>.4891 > 12.129.203.38.http: . ack 4345 win 33304 <nop,nop,timestamp 449964198 164687061> (DF)
> 15 05:20:02.580962 12.129.203.38.http > <me>.4891: . 7241:8689(1448) ack 1615 win 10136 <nop,nop,timestamp 164687071 449964198> (DF)
> 16 05:20:02.581628 <me>.4891 > 12.129.203.38.http: . ack 4345 win 33304 <nop,nop,timestamp 449964288 164687061> (DF)
> 17 05:20:02.581839 truncated-ip - 434 bytes missing! 12.129.203.38.http > <me>.4891: P 8689:10137(1448) ack 1615 win 10136 <nop,nop,timestamp 164687071 449964198> (DF)
> 18 05:20:07.060856 12.129.203.38.http > <me>.4891: . 4345:5793(1448) ack 1615 win 10136 <nop,nop,timestamp 164687519 449964198> (DF)
> 19 05:20:07.061557 <me>.4891 > 12.129.203.38.http: . ack 8689 win 31132 <nop,nop,timestamp 449968770 164687519> (DF)
> 20 05:20:07.061997 <me>.4891 > 12.129.203.38.http: . ack 8689 win 33180 <nop,nop,timestamp 449968770 164687519> (DF)
> 21 05:20:07.144915 12.129.203.38.http > <me>.4891: . 8689:10137(1448) ack 1615 win 10136 <nop,nop,timestamp 164687527 449968770> (DF)
> 22 05:20:07.146198 12.129.203.38.http > <me>.4891: . 10137:11585(1448) ack 1615 win 10136 <nop,nop,timestamp 164687527 449968770> (DF)
> 23 05:20:07.159433 <me>.4891 > 12.129.203.38.http: . ack 11585 win 32580 <nop,nop,timestamp 449968867 164687527> (DF)

The delay looks like a TCP retransmit timeout, but I don't think it
should have happened.  Three identical ACKs (lines 11,14,16) should
have signaled a dropped packet and the sender should have immediately
resent.  Upgrading to FreeBSD 5.3 will help a bit because it can do TCP
SACK, but the cablemodem shouldn't be sending these truncated packets
anyhow.  It's capable of sending full-size packets so it doesn't seem
to be an MTU issue.  Check for updated firmware, maybe?

> More info on my topology:  DSL was PPPoE straight into the fbsd box
> via a crossed Ethernet cable; the cable modem is connected via the
> same cable and uses DHCP.  Fwiw, the NAT is served to Windows boxen
> from a second NIC in the fbsd box.  The drivers for the two NICs are
> ed for the Internet side and dc for the local side.  For PPPoE, I was
> using mpd.  I use ipfw/natd for firewalling and NAT.  I can surely
> provide more details to anyone who needs them.

-- 
	Dan Nelson
	dnelson at allantgroup.com