Rewritten TCP reassembly

[Andrew Gallatin]

Andre Oppermann writes:
 > I've totally rewritten the TCP reassembly function to be a lot more
 > efficient.  In tests with normal bw*delay products and packet loss
 > plus severe reordering I've measured an improvment of at least 30% in
 > performance.  For high and very high bw*delay product links the
 > performance improvement is most likely much higher.
 > 

I ran netperf with 20 times for each of 3 socket buffer sizes (128KB,
256KB, and 1MB) before and after patching.  All tests were run with
net.isr.enable=1, and machdep.cpu_idle_hlt=0.  CPU was pretty much
maxed thoughout. (SMP kernel, 1 HTT p4).

I'm not sure how much weight you should give these result.  I still
haven't gotten around to fixing kttcp, so the memory copy overhead is
pretty high.  For xmit, b/w goes from 2.90Gb/sec to 3.95Gb/sec when
sendfile is used.  (linux receiver on same hw).

I've also not gotten around to making any of Robert's suggested driver
optimizations.

I noticed that at least systat was not printing the number of tcp
out-of-order packets after applying the patch.  I forgot to check
netstat itself.

Drew


x before.131072
+ after.131072
+--------------------------------------------------------------------------+
|                    ++  x  +                        x                    x|
|+    *+  +        + ++ +xx++  x +x*  ++ +  xx+x x   x +  x     x      xxxx|
|           |_____________AM|___________|_______AM__________________|      |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       3336.63       3516.95       3449.87      3447.838     52.708815
+  20          3323       3466.34       3391.97      3389.686     37.154513
Difference at 95.0% confidence
        -58.152 +/- 29.1859
        -1.68662% +/- 0.846499%
        (Student's t, pooled s = 45.5998)


x before.262144
+ after.262144
+--------------------------------------------------------------------------+
|            +                                                             |
|            +                                            x                |
|+      ++ + ++                                        x  x      x         |
|+   + ++++++++      +           +         x  x x x    x xx xxx  xxxx x   x|
|   |______A______|                                |_______AM_______|      |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       3197.77       3421.67       3316.64     3314.8555     59.787371
+  20       2889.64       3122.17       2965.01     2964.9865     50.076867
Difference at 95.0% confidence
        -349.869 +/- 35.2961
        -10.5546% +/- 1.06479%
        (Student's t, pooled s = 55.1463)


x before.1048576
+ after.1048576
+--------------------------------------------------------------------------+
|                 x                                    +                   |
|xx      x      + x    xx  +  x+x++x    *  x*   x+*x+x+*    x*+  +  + +  ++|
|               |_________________|A____M__________A__M____________|       |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  20       2658.05       2732.45       2706.91     2700.1955     23.318833
+  20       2676.13       2748.97       2724.07     2719.9665     20.531036
Difference at 95.0% confidence
        19.771 +/- 14.0613
        0.732206% +/- 0.52075%
        (Student's t, pooled s = 21.9692)