Large TCP send socket buffer optimizations

[Robert Watson]

On Thu, 1 Mar 2007, Andre Oppermann wrote:

> With the TCP socket buffer autosizing and generally larger socket buffers 
> for high bandwidth and high delay connections tcp_output() has become 
> increasingly inefficient for sending segments.  For every segment sent it is 
> traversing the entire socket buffer mbuf chain until it finds the offset to 
> continue from.  Usually this is close to the end of the chain.  Once it got 
> past a few dozen mbufs it starts to bust the CPU caches and performance 
> starts to fall off.
>
> This patch solves the problem by maintaining an offset pointer in the socket 
> buffer to give tcp_output() the closest mbuf right away avoiding the 
> traversal from the beginning.
>
> With this patch we should be able to compete nicely for the Internet land 
> speed record again.
>
> The patch is here:
>
> http://people.freebsd.org/~andre/sockbuf_sndptr-20070301.diff
>
> Any testing, especially on 10Gig cards, and feedback appreciated.

I'm going to be on- and off-line from now through about the last week of March 
on travel to LinuxForum.dk, AsiaBSDCon, and UKUUG.  However, once I get back 
online from that in late March through the end of April, I will be spending 
quite a bit of time looking at high performance packet capture issues, so can 
certainly spend some time looking at this.  Per our earlier discussions and my 
previous exploration of this issue, I see precisely this sort of problem with 
TCP as socket buffer size grows, so should be set up to fairly easily test for 
an improvement once I have my feet on the ground.  If I'm lucky I may get a 
breather for a few days from about 15 March - 18 March and could take an early 
look at it then.

Thanks,

Robert N M Watson
Computer Laboratory
University of Cambridge