TCP packet out-of-order problem

[Václav Haisman]

Lin Jui-Nan Eric wrote, On 30.12.2008 21:06:
> Dear listers,
> 
> We recently found our new FreeBSD server (located in some foreign
> region) has poor network performance. After doing some tcpdump and
> iperf testing, we found that out-of-order TCP packets are not inserted
> into queue.
> 
> This is an 100Mbps line, and TSO is disabled.
> 
> % uname -a
> FreeBSD bsd 7.1-RC2 FreeBSD 7.1-RC2 #2: Wed Dec 31 03:12:39 CST 2008
>   root at bsd:/usr/obj/usr/src/sys/KERNEL  amd64
> 
> % iperf -c 10.1.1.250
> ------------------------------------------------------------
> Client connecting to office, TCP port 5001
> TCP window size: 3.07 MByte (default)
> ------------------------------------------------------------
> [  4] local 10.1.1.210 port 61488 connected with 10.1.1.250 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.2 sec  5.74 MBytes  4.74 Mbits/sec
> 
> 03:47:21.146397 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 159305:160753(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.146409 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 160753
> win 12568 <nop,nop,timestamp 3216612488 555928950>
> 03:47:21.146473 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 160753:162201(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.146485 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201
> win 12568 <nop,nop,timestamp 3216612489 555928950>
> 03:47:21.146972 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 163649:165097(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.146983 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201
> win 12573 <nop,nop,timestamp 3216612489 555928950>
> 03:47:21.146985 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 162201:163649(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.146996 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
> win 12568 <nop,nop,timestamp 3216612489 555928950>
> 03:47:21.146998 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 165097:166545(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.147006 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
> win 12573 <nop,nop,timestamp 3216612489 555928950>
> 03:47:21.147009 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 166545:167993(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 03:47:21.147017 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649
> win 12573 <nop,nop,timestamp 3216612489 555928950>
> 03:47:21.147019 IP 10.1.1.210.54919 > 10.1.1.250.5001: .
> 167993:169441(1448) ack 1 win 1040 <nop,nop,timestamp 555928950
> 3216612488>
> 
> * You can see "ack 163649" repeating, but the packet is transmitted
> before 163649:165097.
> 
> % cat /etc/sysctl.conf
> # $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $
> #
> #  This file is read when going to multi-user and its contents piped thru
> #  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
> #
> 
> # Uncomment this to prevent users from seeing information about processes that
> # are being run under another UID.
> #security.bsd.see_other_uids=0
> debug.bootverbose=1
> kern.ipc.somaxconn=8192
> kern.maxfiles=65536
> kern.maxfilesperproc=32768
> kern.maxprocperuid=65536
> net.inet.ip.fastforwarding=1
> net.inet.tcp.delayed_ack=0
> vm.pmap.shpgperproc=2000
> kern.ipc.maxsockbuf=8388608
> net.inet.tcp.sendspace=3217968
> net.inet.tcp.recvspace=3217968
> 
> Is our configuration wrong? Or it is an known bug? I have searched
> stable & net list, but found no similar discussion.
> Thank you all in advance!
Do you have net.inet.tcp.sack.enable=1?

--
VH