TCP out-of-order packets.

[Julian Elischer]

Brooks Davis wrote:

>On Thu, Jan 13, 2005 at 11:00:52AM -0800, Julian Elischer wrote:
>  
>
>>Brooks Davis wrote:
>>
>>    
>>
>>>On Wed, Jan 12, 2005 at 05:07:36PM -0800, Julian Elischer wrote:
>>>
>>>
>>>      
>>>
>>>>I have a link which is provided by someone else that is 7 x E1s 
>>>>aggregated.
>>>>At leat it looks that way to me when I get to see it. however I have 
>>>>only been able to get
>>>>60kB.sec across this, despite having a tcp window size of 131072 bytes..
>>>>After investigation it appears that the link is massively re-orderring 
>>>>packets.
>>>>groups of upto 10 packets may appear in random order. (Maybe more, bu tI 
>>>>have seen 10)
>>>>
>>>>in fact packets are rarely IN order.
>>>>
>>>>This plays havoc with the tcp sessions.
>>>>
>>>>I was thinking of writing a hacked up version of NATD that
>>>>instead of doing NAT, just did a pre-sort on packets from each session, 
>>>>so that the receiver would
>>>>see a stream of IN-order packets, with occasional delays.
>>>>
>>>>firstly, does anyone have any tools to do this already (why build when 
>>>>you can borrow)
>>>>and secondly, does anyone have any experience with this sort of problem?
>>>>
>>>>I have no control over or access to the link.. all I have is a promise 
>>>>that they will deliver
>>>>14Mb/Sec. with approc 300mSec. RTT to me but there is no promise about 
>>>>packet order.
>>>>
>>>>I just get a 100Mb ethernet cable.
>>>>  
>>>>
>>>>        
>>>>
>>>Have you tried Andre's TCP reassembly rewrite?  He says he saw
>>>significant improvements in the face of major reordering.
>>>
>>>
>>>      
>>>
>>I don't think it's a problem with reassembly overhead, but rather a 
>>symptom of sender
>>backoff when confronted with multiple duplicate acks due to the receiver
>>getting the packets out of order.
>>
>>I wonder if there's a way to turn off the sender backoff?
>>
>>    
>>
>>>http://www.mail-archive.com/freebsd-net@freebsd.org/msg14064.html
>>>
>>>      
>>>
>>These machines are production machines on a custommer site.. running 4.8
>>it would be significant work to put the rewrite in (to 4.8)  and a lot 
>>of red tape to
>>reboot one to the new kernel.. :-/
>>    
>>
>
>Hmm, what about using a bridge with pf's TCP normalizer?  You'd probably
>have to raise your socket buffer sizes to handle the added latency (not
>sure how much that is), but that might be an option (assuming adding
>another machine wasn't harder then installing a new kernel).  That code
>is at least already written so you could also potentially crib from it
>for the hacked up natd idea.
>

I have remote access to a handful of machines on a switch.
I do not have access ot the switch or the wiring or the link..

for added points the machines are in India.

>
>-- Brooks
>
>  
>