TCP Performance advice needed [long!]

Ivan Voras ivoras at fer.hr
Mon Feb 13 10:22:09 PST 2006


I'm writing a simple client-server application and I think I have a 
performance problem. The basic communication between the server and the 
client is like this: the server sends messages at most MAX_SIZE in 
length, one by one, and waits until the client acknowleges each message 
before sending the next one. Messages of maximum length (MAX_SIZE) make 
about 75% or more of total messages. Client responses are small and of 
fixed size (<10 bytes). I have TCP_NODELAY turned on on both. All sends 
are done with a blocking send() call of required size. Client's response 
processing time doesn't depend notably on the message size.

I've done some simple benchmarks and I'm puzzled about the results. In 
the benchmarks, the MAX_SIZE is periodically increased from 256 to 65536 
in power-of-2 steps. In the benchmark setup the server and client are 
connected directly via 100Mbps connection without transmission problems 
(FTP pushes ~12MB/s in both directions).

Problem: It seems that the packets-per-second rate sent by the server 
increases from 1000 at MAX_SIZE=256 to 2500 at MAX_SIZE=4096, then 
sharply drops to 400 at MAX_SIZE=8192 and above. Remembering a 
discussion on Samba performance, I tried disabling tcp.inflight.enable, 
and the results for large packet sizes became very good, about 4000pps, 
until MAX_SIZE=32768, where it drops to 400 again. What does 
tcp.inflight do, and can it be done without changing sysctls?

Another problem: the starting rates, e.g. ~1000pps at MAX_SIZE=256, seem 
too low to me - this pushes only 256kB/s. This is a problem because in 
real usage such small sizes could be common. Looking at netstat, packet 
rates seem to vary randomly. Here's netstat log for session with 
MAX_SIZE=512:


          1     0         66          1     0        178     0
        378     0      27956        378     0     132192     0
       1405     0     103962       1405     0     488770     0
       1193     0      88274       1193     0     414994     0
       1093     0      80874       1093     0     380194     0
       1010     0      74732       1011     0     351658     0
        930     0      68812        930     0     323732     0
        797     0      58970        796     0     276576     0
         95     0       7022         95     0      32890     0
             input        (Total)           output
    packets  errs      bytes    packets  errs      bytes colls
        201     0      14866        201     0      69778     0
        495     0      36622        495     0     172202     0
        739     0      54678        739     0     257002     0
       1291     0      95526       1292     0     449728     0
       1134     0      83908       1133     0     394094     0
       1026     0      75916       1026     0     356616     0
        987     0      73030        987     0     343306     0
        873     0      64594        873     0     303634     0
        855     0      63262        855     0     297370     0
        360     0      26632        360     0     125372     0
       1002     0      74140       1002     0     348264     0
       1166     0      86276       1166     0     405860     0
       1056     0      78136       1056     0     367056     0
        937     0      69330        937     0     325906     0
        753     0      55714        753     0     261874     0
        111     0       8206        111     0      38458     0
        143     0      10566        144     0      49692     0
          1     0         66          1     0        178     0
          1     0         66          1     0        178     0

Here's for MAX_SIZE=8192:

             input        (Total)           output
    packets  errs      bytes    packets  errs      bytes colls
       1082     0      74292       1438     0    1580040     0
       1287     0      88366       1701     0    1885062     0
       1403     0      96342       1867     0    2044074     0
       1763     0     121046       2346     0    2578684     0
       1532     0     105208       2026     0    2236048     0
       1866     0     128124       2463     0    2726554     0
       2401     0     164874       3192     0    3508384     0
       2797     0     192058       3719     0    4095824     0
       2791     0     191646       3714     0    4070142     0
       2773     0     190410       3692     0    4052512     0
       2583     0     177358       3438     0    3780296     0
       2947     0     202358       3927     0    4307032     0
       2963     0     203462       3943     0    4324596     0
       2946     0     202292       3926     0    4307032     0
       2965     0     203594       3946     0    4341638     0
       2989     0     205242       3977     0    4359552     0
       2928     0     201056       3898     0    4289006     0
       2995     0     205654       3983     0    4368192     0
       2965     0     203594       3945     0    4333018     0
       2983     0     204830       3973     0    4359598     0
       2978     0     204484       3968     0    4359380     0
             input        (Total)           output
    packets  errs      bytes    packets  errs      bytes colls
       2986     0     205028       3976     0    4359910     0
       2979     0     204566       3969     0    4359578     0
       2123     0     145790       2827     0    3089432     0

It seems that the transmission for MAX_SIZE=8192 has slow start but I 
don't know how to disable it?

Help on any of these points, or suggestions where to investigate further 
will be appreciated!


More information about the freebsd-net mailing list