Slow speeds experienced with Dummynet

[rihad]

Luigi Rizzo wrote:
> On Fri, Feb 19, 2010 at 10:48:32PM +0400, rihad wrote:
>> Hi, all,
>>
>> Recalling my old posting "dummynet dropping too many packets" dated 
>> October 4, 2009, the problem isn't over just yet. This time, there are 
>> no interface i/o drops (just a reminder: we have 2 bce(4) GigE cards 
>> connected to a Cisco router, one for input, and one for output. The box 
>> itself does some traffic accounting and enforces speed limits w/ 
>> ipfw/dummynet. There are normally around 5-6k users online).
> 
> If i remember well, the previous discussion ended when you
> raised the intr_queue_maxlen (and perhaps increased HZ) to
> avoid that the bursts produced by periodic invocation of
> dummynet_io() could overflow that queue.
> 
>>From the rest of your post it is not completely clear if you have
> not found any working configuration, or there is some setting (e.g.
> with "queue 1000" or larger) which do produce a smooth experience
> for your customers.
> 
Nope, I haven't been able to find it :( From as low as 50 to 2000 slots. 
I've also tried 1 and 2s worth queue sizes (in Kbytes). After about 8 
p.m., when most users are online, speed problems are the most apparent. 
It all boils down to big differences between some subsequent "systat 
-if" reads, both for input and output, when dummynet is in use. It isn't 
normal for two reads to differ in as much as 100-150 mbps (20-25%). I 
can only hope that Intel's expensive 10 GigE cards have much larger 
tolerance to traffic load spikes, and are better suited for ISP usage 
patterns.

> Another thing i'd like to understand is whether all of your pipes
> have a /32 mask, or there are some which cover multiple hosts.
> Typical TCP connections have around 50 packets in flight when the
> connection is fully open (which in turn is hard to happen on a 512k
> pipe) so a queue of 100-200 is unlikely to overflow.
> 
> In fact, long queues are very detrimental for customers because
> they increase the delay of the congestion control loop -- as a rule
> of thumb, you should try to limit the queue size to at most 1-2s
> of data.
> 
> cheers
> luigi
> 
>> Traffic shaping is accomplished by this ipfw rule:
>> pipe tablearg ip from any to table(0) out
>> where table(0) contains those 5-6k IP addresses. The pipes themselves 
>> are GRED (or taildrop, it doesn't matter):
>> ipfw pipe   512 config bw   512kbit/s mask dst-ip 0xffffffff gred 
>> 0.002/900/1000/0.1 queue 1000
>> Taking this template the speeds range from 512 to tens of mbps. With the 
>> setup as above very many users complain about very slow downloads, slow 
>> browsing. systat -ifstat, refreshed every 5 seconds, does reveal large 
>> differences between subsequent displays: if around 800-850 mbps is 
>> what's to be expected, it doesn't stay within those limits, also jumping 
>> to as low as 620+, and to somewhere in the 700's, Now imagine this: once 
>> I turn dummynet off (by short-circuiting a "allow ip from any to any" 
>> before the "pipe tablearg" rule) all user complaints stop, with traffic 
>> load staying stably at around 930-950 mbps.
>>
>> Does this have anything to do with "dummynet bursts"? How can I beat 
>> that? If I keep the pipe queue size at 2000 slots, the 
>> net.inet.ip.dummynet.io_pkt_drops sysctl stops increasing, once I start 
>> tweaking the value to as low as 100 slots, the value starts raising 
>> constantly at about 300-500 pps. I had hoped that smaller queue sizes 
>> would battle the negative effects of dummynet burstiness, it did so, I 
>> guess, but not in a very decisive manner.
>>
>>
>> FreeBSD 7.1-RELEASE-p10
>> kern.hz=4000
>> kern.ipc.nmbclusters=111111
>> net.inet.ip.fastforwarding=1
>> net.inet.ip.dummynet.io_fast=1
>> net.isr.direct=0
>> net.inet.ip.intr_queue_maxlen=5000
>> net.inet.ip.dummynet.hash_size=512
>> net.inet.ip.dummynet.max_chain_len=16
>>
>> net.inet.ip.intr_queue_drops: 0
>> systat -ip shows zero output drops at times of trouble. netstat -s's 
>> "output packets dropped due to no bufs, etc." is also fine. netstat -m 
>> shows nothing suspicious.
>>
>> p.s: Two "bloody expensive" Intel 10 GigE's are on their way to us to 
>> replace the Broadcom cards, meanwhile what should I try doing? Thanks 
>> for reading.
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> 
>