[fbsd] Re: [fbsd] Network performance in a dual CPU system

[Robert Watson]

On Tue, 2 May 2006, Brian Candler wrote:

> On Mon, May 01, 2006 at 11:38:39AM +1000, lukem.freebsd at cse.unsw.edu.au wrote:
>> Would it be possible to improve the behaviour of the TCP protocol
>> implementation so that out-of-order reception was acceptable?
>
> Possibly - but if your FreeBSD box is acting as a router, and it re-orders 
> packets in transit to the rest of the Internet, then in principle you'd need 
> to upgrade the TCP implementations on every other device in the Internet :-(

The behavior in question is referred to as a fast retransmit. The basic idea 
is this: if three datagrams are sent by the remote stack in order, and you 
receive two datagrams with a clear "missing" middle one, then you can assume 
that the middle one went missing since network topologies have been carefully 
crafted to meet TCP's assumptions of ordering.  You hint to the remote stack 
this has happened by ACK'ing the first segment twice.  (I may have the details 
not quite right here, but the effect is the same).  Life gets a bit better 
with SACK.

Changing this assumption is very difficult -- it's not a property purely of 
our implementation, as much as a property of the wire protocol, and written 
into the standards.  Even if we don't play by that rule, other stacks will. 
In particular, this is an important case when you're routing with FreeBSD -- 
if you reorder packets in the routing path, end hosts on either side of it 
will get very upset.  In this case, FreeBSD isn't even a direct party to TCP, 
and so can't change the behavior.

This isn't just a property of TCP, many protocols perform badly in the 
presence of reordering, and so it's really best to avoid it wherever possible, 
and we go to significant lengths to do so.  The trick is to work away at the 
edges and try to (over time) move towards the minimal ordering constraints.

Robert N M Watson