TCP out-of-order packets.

[Brooks Davis]

On Thu, Jan 13, 2005 at 11:00:52AM -0800, Julian Elischer wrote:
> 
> 
> Brooks Davis wrote:
> 
> >On Wed, Jan 12, 2005 at 05:07:36PM -0800, Julian Elischer wrote:
> > 
> >
> >>I have a link which is provided by someone else that is 7 x E1s 
> >>aggregated.
> >>At leat it looks that way to me when I get to see it. however I have 
> >>only been able to get
> >>60kB.sec across this, despite having a tcp window size of 131072 bytes..
> >>After investigation it appears that the link is massively re-orderring 
> >>packets.
> >>groups of upto 10 packets may appear in random order. (Maybe more, bu tI 
> >>have seen 10)
> >>
> >>in fact packets are rarely IN order.
> >>
> >>This plays havoc with the tcp sessions.
> >>
> >>I was thinking of writing a hacked up version of NATD that
> >>instead of doing NAT, just did a pre-sort on packets from each session, 
> >>so that the receiver would
> >>see a stream of IN-order packets, with occasional delays.
> >>
> >>firstly, does anyone have any tools to do this already (why build when 
> >>you can borrow)
> >>and secondly, does anyone have any experience with this sort of problem?
> >>
> >>I have no control over or access to the link.. all I have is a promise 
> >>that they will deliver
> >>14Mb/Sec. with approc 300mSec. RTT to me but there is no promise about 
> >>packet order.
> >>
> >>I just get a 100Mb ethernet cable.
> >>   
> >>
> >
> >Have you tried Andre's TCP reassembly rewrite?  He says he saw
> >significant improvements in the face of major reordering.
> > 
> >
> 
> I don't think it's a problem with reassembly overhead, but rather a 
> symptom of sender
> backoff when confronted with multiple duplicate acks due to the receiver
> getting the packets out of order.
> 
> I wonder if there's a way to turn off the sender backoff?
> 
> >http://www.mail-archive.com/freebsd-net@freebsd.org/msg14064.html
> >
> 
> These machines are production machines on a custommer site.. running 4.8
> it would be significant work to put the rewrite in (to 4.8)  and a lot 
> of red tape to
> reboot one to the new kernel.. :-/

Hmm, what about using a bridge with pf's TCP normalizer?  You'd probably
have to raise your socket buffer sizes to handle the added latency (not
sure how much that is), but that might be an option (assuming adding
another machine wasn't harder then installing a new kernel).  That code
is at least already written so you could also potentially crib from it
for the hacked up natd idea.

-- Brooks

-- 
Any statement of the form "X is the one, true Y" is FALSE.
PGP fingerprint 655D 519C 26A7 82E7 2529  9BF0 5D8E 8BE9 F238 1AD4
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20050113/82026d16/attachment.bin