Issues with a Large Fat pipe Network simulation

[Pieter de Boer]

Luigi Rizzo wrote:

> oh yes one thing... you are using 'via foo0' in your rule,
> which means the packet is intercepted both in the input and
> output path, which causes further contention on the queues.
Well, when using 'ip from client to server recv em0', packets get 
matched twice. When I set some latency on that pipe, the packets incur 
double the latency I set. With 'via em0' this isn't the case. I tried it 
out, but didn't seem to make much difference.

> also you can set the queue size in kbytes, and can probably go as high
> as 1000kb. maybe this helps too.
Doesn't seem to do much either.

> I am pretty sure there is some issue there, also related to some
> timing issues and tcp window opening mode (slow start vs. linear)
I went to see if there were any sysctl's I could tune a bit. I found these:
net.inet.ip.intr_queue_maxlen: 50
net.inet.ip.intr_queue_drops: 382136

I don't like drops. So I set intr_queue_maxlen to 400, and -poof-, the 
speed went up to around 700mbit/s. Still not as fast as it was with 64KB 
send/recv spaces, but it's a huge improvement nonetheless.

I guess we probably should tune a bit more until we're confident that 
the middle-box behaves correctly, before adding things like latency and 
packet-loss :)

Thanks for the advice! If you know other settings to tune on the 
dummynetting host, I'd very much like to hear them. I'm pondering about 
polling (which means we can't do SMP on the dummynet system, but it's 
only pushing packets, so that shouldn't matter too much). At least we 
have somewhat more info to work with now :)

-- 
Pieter