TSO and FreeBSD vs Linux

[Julian Elischer]

On 8/14/13 1:29 AM, Julian Elischer wrote:
> I have been tracking down a performance embarrassment on AMAZON EC2 
> and have found it I think.
> Our OS cousins over at Linux land have implemented some interesting 
> behaviour when TSO is in use.
>
> They seem to aggregate ACKS when there is a lot of traffic so that 
> they can create the
> largest possible TSO packet. We on the other hand respond to each 
> and every returning ACK, as it arrives and thus generally fall into 
> the behaviour of sending a bunch of small packets, the size of each 
> ack.
>
> for two examples look at:
>
>
> http://www.freebsd.org/~julian/LvsF-tcp-start.tiff
> and
> http://www.freebsd.org/~julian/LvsF-tcp.tiff

some people have troubel with tiff, so here they are a jpeg.

http://www.freebsd.org/~julian/LvsF-tcp-start.jpg
and
http://www.freebsd.org/~julian/LvsF-tcp.jpg
>
> in each case, we can see FreeBSD on the left and Linux on the right.
>
> The first case shows the case as the sessions start, and the second 
> case shows
> some distance later (when the sequence numbers wrap around.. no 
> particular
> reason to use that, it was just fun to see).
> In both cases you can see that each Linux packet (white)(once they 
> have got
> going) is responding to multiple bumps in the send window sequence
> number (green and yellow lines) (representing the arrival of several 
> ACKs)
> while FreeBSD produces a whole bunch of smaller packets, slavishly 
> following
> exactly the size of each incoming ack.. This gives us quite  a 
> performance debt.
> Notice that this behaviour in Linux seems to be modal.. it seems to 
> 'switch on' a little bit
> into the 'starting' trace.
>
> In addition, you can see also that Linux gets going faster even in 
> the beginning where
> TSO isn't in play, by sending a lot more packets up-front. (of 
> course the wisdom of this
> can be argued).
>
> Has anyone done any work on aggregating ACKs, or delaying responding 
> to them?
>
> Julian
> (Who's suspecting he's about to find out more about TSO and the send 
> path, than he ever wanted to).
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
>