Rewritten TCP reassembly

[Andre Oppermann]

Andrew Gallatin wrote:
> 
> Andre Oppermann writes:
>  > I've totally rewritten the TCP reassembly function to be a lot more
>  > efficient.  In tests with normal bw*delay products and packet loss
>  > plus severe reordering I've measured an improvment of at least 30% in
>  > performance.  For high and very high bw*delay product links the
>  > performance improvement is most likely much higher.
>  >
> 
> I ran netperf with 20 times for each of 3 socket buffer sizes (128KB,
> 256KB, and 1MB) before and after patching.  All tests were run with
> net.isr.enable=1, and machdep.cpu_idle_hlt=0.  CPU was pretty much
> maxed thoughout. (SMP kernel, 1 HTT p4).

HTT is not really good with FreeBSD's schedulers.  Most of the time
it hurts performance more than it helps.

> I'm not sure how much weight you should give these result.  I still
> haven't gotten around to fixing kttcp, so the memory copy overhead is
> pretty high.  For xmit, b/w goes from 2.90Gb/sec to 3.95Gb/sec when
> sendfile is used.  (linux receiver on same hw).

Ok.

> I've also not gotten around to making any of Robert's suggested driver
> optimizations.

Ok.

> I noticed that at least systat was not printing the number of tcp
> out-of-order packets after applying the patch.  I forgot to check
> netstat itself.

Duh, that's missing in the patch.  Good you noticed.

Regarding your measurements, did you measure the bandwidth as reported
by Netperf?  Is a FreeBSD box on both sides (you mentioned Linux)?

-- 
Andre


> Drew
> 
> x before.131072
> + after.131072
> +--------------------------------------------------------------------------+
> |                    ++  x  +                        x                    x|
> |+    *+  +        + ++ +xx++  x +x*  ++ +  xx+x x   x +  x     x      xxxx|
> |           |_____________AM|___________|_______AM__________________|      |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       3336.63       3516.95       3449.87      3447.838     52.708815
> +  20          3323       3466.34       3391.97      3389.686     37.154513
> Difference at 95.0% confidence
>         -58.152 +/- 29.1859
>         -1.68662% +/- 0.846499%
>         (Student's t, pooled s = 45.5998)
> 
> x before.262144
> + after.262144
> +--------------------------------------------------------------------------+
> |            +                                                             |
> |            +                                            x                |
> |+      ++ + ++                                        x  x      x         |
> |+   + ++++++++      +           +         x  x x x    x xx xxx  xxxx x   x|
> |   |______A______|                                |_______AM_______|      |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       3197.77       3421.67       3316.64     3314.8555     59.787371
> +  20       2889.64       3122.17       2965.01     2964.9865     50.076867
> Difference at 95.0% confidence
>         -349.869 +/- 35.2961
>         -10.5546% +/- 1.06479%
>         (Student's t, pooled s = 55.1463)
> 
> x before.1048576
> + after.1048576
> +--------------------------------------------------------------------------+
> |                 x                                    +                   |
> |xx      x      + x    xx  +  x+x++x    *  x*   x+*x+x+*    x*+  +  + +  ++|
> |               |_________________|A____M__________A__M____________|       |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x  20       2658.05       2732.45       2706.91     2700.1955     23.318833
> +  20       2676.13       2748.97       2724.07     2719.9665     20.531036
> Difference at 95.0% confidence
>         19.771 +/- 14.0613
>         0.732206% +/- 0.52075%
>         (Student's t, pooled s = 21.9692)