Rewritten TCP reassembly

[Andrew Gallatin]

Andre Oppermann writes:
 > 
 > I've got some excellent review feedback from Mike Spengler and he found
 > a off-by-one queue limit tracking error.
 > 
 >   http://www.nrg4u.com/freebsd/tcp_reass-20041213.patch
 > 


Here are the same tests running your new patch in comparison to a
stock 6.x kernel from Friday.  It looks like everything is faster.
I've never seen 3.7Gb/sec on FreeBSD before.  Nice work!

The "before" and "tcp_reass-20041213" kernels differ only in the
contents of the above patch.  I ran netperf with 20 times for each of
4 socket buffer sizes (64KB, 128KB, 256KB, and 1MB) before and after
patching.  All tests were run with net.isr.enable=1, and
machdep.cpu_idle_hlt=0.  CPU was pretty much maxed thoughout. (SMP
kernel, 1 HTT p4).  All the numbers in Mb/sec.  The sender was running
linux-2.6.6 (also SMP on an identical HTT P4).

The out-of-order accounting bug seems to be gone.


Drew


x before.65536
+ tcp_reass-20041213.65536
+--------------------------------------------------------------------------+
|                                  +                                       |
|                                  +                                       |
|  xx           +  +        +  +   +*x  +   ++   ++ +   ++          +   +++|
||__________________A________|______M__|_____MA________________|           |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   4       3121.03       3135.88       3135.27     3128.3325     8.3667691
+  20       3126.75       3151.97       3139.41      3139.608     7.5192088
Difference at 95.0% confidence
	11.2755 +/- 8.67923
	0.360432% +/- 0.277439%
	(Student's t, pooled s = 7.64032)
x before.131072
+ tcp_reass-20041213.131072
+--------------------------------------------------------------------------+
|          x             x   ++    xx           +               +  +       |
|x         x  xxx   xx*x x  x+* + xxx++  +   +  +  ++ +         +  +      +|
|           |_________AM_______||_____________A_M_____________|            |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       3336.63       3516.95       3449.87      3447.838     52.708815
+  20       3443.67       3717.15       3579.92     3571.8355     80.445555
Difference at 95.0% confidence
	123.997 +/- 43.5271
	3.59638% +/- 1.26245%
	(Student's t, pooled s = 68.0063)
x before.262144
+ tcp_reass-20041213.262144
+--------------------------------------------------------------------------+
|                                    x       +                             |
|                                    x    x ++  x    +   ++                |
|+            x    x  x x       xx+ *x x  * ++ +xx xx+  x++ + *  +        +|
|                         |_______|____A_________A__|M__________|          |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       3197.77       3421.67       3316.64     3314.8555     59.787371
+  20       3135.57       3479.29       3380.31     3361.5785     71.303501
Difference at 95.0% confidence
	46.723 +/- 42.1136
	1.4095% +/- 1.27045%
	(Student's t, pooled s = 65.7979)
x before.1048576
+ tcp_reass-20041213.1048576
+--------------------------------------------------------------------------+
|                                   +               +                      |
|              x         +          +x     x+      x+                      |
|xx     x      x    *x   * x x    * +x  +* x+x+x++ x+   ++                +|
|             |_______________A|__M________AM|__________|                  |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       2658.05       2732.45       2706.91     2700.1955     23.318833
+  20       2686.35       2765.88       2722.09     2720.7805     18.752084
Difference at 95.0% confidence
	20.585 +/- 13.5427
	0.762352% +/- 0.501546%
	(Student's t, pooled s = 21.159)