TSO and FreeBSD vs Linux

[Julian Elischer]

I have been tracking down a performance embarrassment on AMAZON EC2 
and have found it I think.
Our OS cousins over at Linux land have implemented some interesting 
behaviour when TSO is in use.

They seem to aggregate ACKS when there is a lot of traffic so that 
they can create the
largest possible TSO packet. We on the other hand respond to each and 
every returning ACK, as it arrives and thus generally fall into the 
behaviour of sending a bunch of small packets, the size of each ack.

for two examples look at:


http://www.freebsd.org/~julian/LvsF-tcp-start.tiff
and
http://www.freebsd.org/~julian/LvsF-tcp.tiff

in each case, we can see FreeBSD on the left and Linux on the right.

The first case shows the case as the sessions start, and the second 
case shows
some distance later (when the sequence numbers wrap around.. no particular
reason to use that, it was just fun to see).
In both cases you can see that each Linux packet (white)(once they 
have got
going) is responding to multiple bumps in the send window sequence
number (green and yellow lines) (representing the arrival of several ACKs)
while FreeBSD produces a whole bunch of smaller packets, slavishly 
following
exactly the size of each incoming ack.. This gives us quite  a 
performance debt.
Notice that this behaviour in Linux seems to be modal.. it seems to 
'switch on' a little bit
into the 'starting' trace.

In addition, you can see also that Linux gets going faster even in the 
beginning where
TSO isn't in play, by sending a lot more packets up-front. (of course 
the wisdom of this
can be argued).

Has anyone done any work on aggregating ACKs, or delaying responding 
to them?

Julian
(Who's suspecting he's about to find out more about TSO and the send 
path, than he ever wanted to).