SACK (and PF) wierdness

[Daniel Hartmeier]

Pawel kindly supplied a tcpdump of an entire connection. We identified
the point where pf drops the packet because it sees a violation of the
advertised sequence window.

I think we can show that there is some problem with the TCP code,
possibly related to SACK. Hence, I'd like to ask anyone familar with TCP
and SACK to help.

You can find the output on

  http://www.benzedrine.cx/pawel.txt (328k)

This is an active FTP data connection, from FTP server 192.168.1.10:20
to FTP client 192.168.1.200:64828, where payload is only flowing from
server to client. Both hosts run recent FreeBSD 6-current.

The dump consists of 1571 packets, 940 from server to client and 631
from client to server. pf complains about the 1572th packet (not found
in the dump, as it was blocked):

Nov 22 23:25:20 <kern.crit> darkstar kernel: pf: BAD state: TCP
192.168.1.10:20 192.168.1.10:20 192.168.1.200:64828
[lo=1185380879 high=1185380879 win=33304 modulator=0 wscale=1]
[lo=1046638749 high=1046705357 win=33304 modulator=0 wscale=1]
4:4 A seq=1185380879 ack=1046638749 len=1448 ackskew=0 pkts=940:631
dir=out,fwd
Nov 22 23:25:20 <kern.crit> darkstar kernel: pf: State failure on: 1 |

This means the server was trying to send 1448 bytes from th_seq
1185380879, but that violated the upper boundary of what pf thinks the
client may accept. pf calculates this upper boundary from th_ack +
th_win from the client.

Let's look at what the client sent last:

23:25:20.203732 192.168.1.200.64828 > 192.168.1.10.20: . [tcp sum ok]
1046638749:1046638749(0) ack 1185318615 win 31132 <nop,nop,timestamp
3617747809 2499667199,nop,nop,sack 1 {1185320063:1185369295} > (DF) [tos
0x8] (ttl 64, id 54682, len 64)

Windows scaling is enabled, and both peers' scaling factors are 2^1, so
the upper boundary advertised is

  th_ack 1185318615 + 2 * th_win 31132 == 1185380879

Why does the server try to send 1448 bytes from 1185380879 upwards?
This is a violation of the client's window by the server, right?

If you look at some prior packets from the client, from 23:25:20.145089
to 23:25:20.203732, it sent 33 ACKs with equal th_ack 1185314271 but
different first SACK confirmations.

We've looked at several examples like this, one common thing was that
the last packet before the drop was always a zero-payload packet from
the server to the client at exactly the client's advertised window
border. Maybe that rings a bell for someone.

Any ideas welcome :)
Daniel