Very high wide area TCP tuning

[Lawrence Stewart]

Hi Kevin,

Kevin Oberman wrote:
> I am attempting to use a FreeBSD box with either a Myricom or Chelsio
> 10GE card to generate very large TCP streams over cross-country
> links. The RTT for the test path is 94 ms. It is dedicated to my testing
> at this time, so I have no contention other than a few KB of routing
> updates.
>
> Clearly, I need a very large window...about 120 MB, but I am unsure how
> FreeBSD will handle this. (Unless I do other things, I suspect it will
> not be pretty.) I imagine I will need a large kvm space, at the least,
> but are there any other sysctls that are likely to need adjusting to
> make this all work? IS it likely to work better on a amd64 system than a
> i386?
>
> Any suggestions would be appreciated.
>   

We recently wrote a small technical report that covers some of the 
issues related to FreeBSD network tuning for some TCP research we've 
been doing.

The report's title is "Tuning and Testing the FreeBSD 6 TCP Stack" and 
you can grab it from here: 
http://caia.swin.edu.au/reports/070717B/CAIA-TR-070717B.pdf

We didn't tune for 10GB speeds, but I imagine the principles should all 
still apply. As you correctly point out, you'll also probably want to 
raise the allowable kernel mem size using the "vm.kmem_size" and 
"vm.kmem_size_max" sysctls to avoid any random kernel panics. We used a 
kmem size of 500MB which worked flawlessly with our GigE multiflow tests 
and left us plenty of room to move. This is something you might have to 
use a bit of trial and error to figure out though to ensure you get 
something stable. And of course you're going to want to control the 
number of flows active at one time based on your kmem setting and 
hardware constraints...

Cheers,
Lawrence

http://caia.swin.edu.au