Major performance hit with ToS setting

[Lawrence Stewart]

On 05/31/12 13:33, Kevin Oberman wrote:
[snip]
> I used SIFTR at the suggestion of Lawrence Stewart who headed the
> project to bring plugable congestion algorithms to FreeBSD and found
> really odd congestion behavior. First, I do see a triple ACK, but the
> congestion window suddenly drops from 73K to 8K. If I understand
> CUBIC, it should half the congestion window, not what is happening..
> It then increases slowly (in slow start) to 82K. while the slow-start
> bytes are INCREASING, the congestion window again goes to 8K while the
> SS size moves from 36K up to 52K. It just continues to bound wildly
> between 8K (always the low point) and between 64k and 82K. The swings
> start at 83K and, over the first few seconds the peaks drop to about
> 64K.

Oh, and a comment about this behaviour. Dropping back to 8k (1MSS) is 
only nasty if the TF_{CONG|FAST}RECOVERY flags are *not* set i.e. if you 
see cwnd grow, drop to 8k with those flags set, and then when the flags 
are unset, cwnd starts at the value of ssthresh, then that is perfectly 
normal recovery behaviour. What *is* nasty is if an RTO fires, which 
will reset cwnd to 8k, ssthresh to 2*MSS and make the connection 
effectively start from scratch again.

There is evidence of RTOs in your siftr output, which is bad news e.g 
here's one example of 2 side-by-side log lines from your trace:

# Direction,time,ssthresh,cwnd,flags
i,1338319593.574706,27044,27044,1630544864
o,1338319593.831482,16384,8192,1092625377

Note the 300ms gap, and how cwnd resets to 1MSS and flags go from 
1630544864 
(TF_WASCRECOVERY|TF_CONGRECOVERY|TF_WASFRECOVERY|TF_FASTRECOVERY) to 
1092625377 (TF_WASCRECOVERY|TF_WASFRECOVERY).

Cheers,
Lawrence