Rewritten TCP reassembly

[Andre Oppermann]

Andrew Gallatin wrote:
> 
> Andre Oppermann writes:
>  >
>  > I've got some excellent review feedback from Mike Spengler and he found
>  > a off-by-one queue limit tracking error.
>  >
>  >   http://www.nrg4u.com/freebsd/tcp_reass-20041213.patch
>  >
> 
> Here are the same tests running your new patch in comparison to a
> stock 6.x kernel from Friday.  It looks like everything is faster.

Excellent!

> I've never seen 3.7Gb/sec on FreeBSD before.  Nice work!

Thanks!

> The "before" and "tcp_reass-20041213" kernels differ only in the
> contents of the above patch.  I ran netperf with 20 times for each of
> 4 socket buffer sizes (64KB, 128KB, 256KB, and 1MB) before and after
> patching.  All tests were run with net.isr.enable=1, and
> machdep.cpu_idle_hlt=0.  CPU was pretty much maxed thoughout. (SMP
> kernel, 1 HTT p4).  All the numbers in Mb/sec.  The sender was running
> linux-2.6.6 (also SMP on an identical HTT P4).
> 
> The out-of-order accounting bug seems to be gone.

Perfect.  I want to get this patch into the tree next.

I have already the next round in the works which is optimized even more
by merging consecutive mbuf chains together (at the moment I have packet
segment chains which have a direct pointer to the mbuf at the end of the
chain) and which get passed in one go to soappend_stream.  This removes
the "present" loop and simplifies the general code a bit more again.

With this and two other optimizations I have in mind you should be able
to get very close to the theoretical maximum bandwidth of your current
4Gig Myrinet cards.

There are a couple of other TCP tweaks that would help your special case
some more now though.

-- 
Andre