On the trail of a dummynet/bridge/ipfw bug.

[Wade Klaver]

PROBLEM DESCRIPTION

I have a bridge set up on a 7.0 box and am attempting to use it to limit
HTTP connections outgoing from a box behind it to 192Kbit/s for testing.
During this testing I ran into some problems.  At first, I found that
the number of simultaneous pipes was limited to 1024, allowing only 1024
192Kbit/s clients.  Additional clients were simply blocked.  I am using
a very simple firewall config:

  ipfw pipe 1 config bw 192Kbits/s mask all
  ipfw add 00051 skipto 99 ip from 192.168.0.0/16 to 192.168.0.0/16
  ipfw add 00052 skipto 1000 ip from any to any
  ipfw add 00100 pipe 1 ip from 192.168.10.88 80 to any via bridge0
  ipfw add 00200 pipe 1 ip from any 25111 to any via bridge

Regardless of how many clients I threw at the box, I had the limit:

[root at ibm3550b ~]# ipfw pipe show | wc -l
    1028

We managed to track this down to a problem in the ipfw2 userland app.
The following patch to /usr/src/sbin/ipfw/ipfw2.c allowed this limit to
be surpassed.  It would appear that ipfw does not dynamically resize the
pipe array beyond the initial 1024 elements allocated.

# diff ipfw2.c ipfw2.c.orig 
2507c2507
<       int nalloc = 8192;      /* start somewhere... */
---
>       int nalloc = 1024;      /* start somewhere... */

However, this just revealed a bigger problem, potentially do to the
above patch, potentially due to something worse.  Now the bridge will
allow more connections, up to around 2300 where the bridge just dies.
and no more traffic passes.  It is worth noting that I can still connect
to the bridge itself if it has an IP assigned to it, but traffic through
the bridge ceases.  It is also remedied by a /etc/rc.d/netif restart.

ADDITIONAL NOTES
Please let me know if there is any additional information I can provide.
In the kernel options below, HZ=2000 was just something I was trying.
The problem exhibits itself with HZ=1000 as well.
I posted this to -hackers and to -ipfw. Please direct me and future correspondence 
on this issue to the most appropriate list.  I just felt it was not
solid enough to go to -bugs yet.

SYSTEM INFORMATION
IBM 3550
XEON 5345
4GB Memory
[root at ibm3550b /usr/src/sys]# uname -a
FreeBSD ibm3550b.itivalabs.net 7.0-STABLE FreeBSD 7.0-STABLE #13: Wed
Mar 12 03:26:08 PDT 2008
root at ibm3550b.itivalabs.net:/usr/obj/usr/src/sys/WADE  amd64

Bridge members:
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xc8000000-0xc9ffffff irq 18 at device 0.0 on pci4
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xce000000-0xcfffffff irq 16 at device 0.0 on pci6

Kernel options:
# Make an SMP-capable kernel by default
options         SMP        # Symmetric MultiProcessor Kernel
options         LIBALIAS
options         IPFIREWALL              #firewall
options         IPFIREWALL_VERBOSE      #enable logging to syslogd(8)
options         IPFIREWALL_VERBOSE_LIMIT=100    #limit verbosity
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by
default
options         IPFIREWALL_FORWARD      #packet destination changes
options         IPFIREWALL_NAT          #ipfw kernel nat support
options         IPDIVERT                #divert sockets
#options         IPFILTER                #ipfilter support
#options         IPFILTER_LOG            #ipfilter logging
#options         IPFILTER_LOOKUP         #ipfilter pools
#options         IPFILTER_DEFAULT_BLOCK  #block all packets by default
options         IPSTEALTH               #support for stealth forwarding
options         MBUF_STRESS_TEST
options         DUMMYNET
options         HZ=2000
options         EXT2FS

/etc/sysctl.conf
net.inet.ip.fw.dyn_max=8192
net.inet.ip.dummynet.hash_size=4096
net.inet.ip.fw.dyn_buckets=1024
net.inet.ip.dummynet.max_chain_len=64
kern.ipc.nmbclusters=32768

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20080312/c5dc3e46/attachment.pgp