dummynet dropping too many packets

[rihad]

Luigi Rizzo wrote:
> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>> Hi, we have around 500-600 mbit/s traffic flowing through a 7.1R Dell 
>> PowerEdge w/ 2 GigE bce cards. There are currently around 4 thousand ISP 
>> users online limited by dummynet pipes of various speeds. According to 
>> netstat -s output around 500-1000 packets are being dropped every second 
>> (this accounts for wasting around 7-12 mbit/s worth of traffic according 
>> to systat -ifstat):
> 
> what kind of packets are you seeing as dropped ?
> please give the output of 'netstat -s output | grep drop'
> 
The output packets, like I said:
# netstat -s output | grep drop
         2 connections closed (including 2 drops)
         0 embryonic connections dropped
                 2 connections dropped by rexmit timeout
                 0 connections dropped by persist timeout
         0 Connections (fin_wait_2) dropped because of timeout
                 0 connections dropped by keepalive
                 0 dropped
         2 dropped due to no socket
         0 dropped due to full socket buffers
         0 fragments dropped (dup or out of space)
         2 fragments dropped after timeout
         7538 output packets dropped due to no bufs, etc.

The statistics are zeroed every 15 seconds in another window as I'm 
investigating the issue, but the rate is around 500-1000 lost packets 
every second at the current ~530 mbit/s load.

> At those speeds you might be hitting various limits with your
> config (e.g. 50k nmbclusters is probably way too small for

I bet it isn't:
1967/5009/6976/50000 mbuf clusters in use (current/cache/total/max)

> 4k users -- means you have an average of 10-15 buffers per user;
> the queue size of 350kbytes = 2.6Mbits means 2.6 seconds of buffering,
> which is quite high besides the fact that in order to scale to 4k users
> you would need over 1GB of kernel memory just for the buffers).
Aha. Can you be more specific about the kernel memory stuff? Which 
setting needs tweaking?


I have another similar box with 2 em GigE interfaces working @220-230 
mbit/s and virtually no out-of-bufs dropped packets as with bce @500-600 
mbit. It too has 350KBytes dummynet queue sizes. And it too has adequate 
mbuf load:
3071/10427/13498/25600 mbuf clusters in use (current/cache/total/max)