[High Interrupt Count] Networking Difficulties

[Paul A. Procacci]

Gents,

I'm having quite an aweful problem that I need a bit of help with.

I have an HPDL360 G3 ( http://h18000.www1.hp.com/products/quickspecs/11504_na/11504_na.HTML ) which acts as a NAT (via PF) for several (600+) class C's amongst 24+ machines sitting behind it.
It's running FPSense (FreeBSD 8.1-RELEASE-p4).

The important guts are:

2 x 2.8 GHz Cpus
2 BGE interfaces on a PCI-X bus.

During peak times this machine is only able to handle between 500Mbps - 600Mbps before running out of cpu capacity.  (300Mbps(ish) on the LAN, 300Mbps(ish) on the WAN) It's due to the high number of interrupts.
I was speaking with a networking engineer here and he mentioned that I should look at "Interrupt Coalescing" to increase throughput.
The only information I found online regarding this was a post from 2 years ago here: http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022227.html

The tunables mentioned in the above post aren't present in my system, so I imagine this never made it into the bge driver.  Assuming this to be the case, I started looking at DEVICE_POLLING as a solution.
I did try implementing device polling, but the results were worse than I expected.  netisr was using 100% of a single cpu while the other cpu remained mostly idle.
Not knowing exactly what netisr is, I reverted the changes.

This leads me to this list.  Given the scenario above, I'm nearly certain I need to use device polling instead of the standard interrupt driven setup.
The two sysctl's that I've come across thus far that I think are what I need are:

net.isr.maxthreads
hern.hz

I would assume setting net.isr.maxthreads to 2 given my dual core machine is advisable, but I'm not 100% sure.
What are the caveats in setting this higher?  Given the output of `sysctl -d net.isr.maxthreads` I would expect anything higher than the number of cores to be detrimental.  Is this correct?

kern.hz I'm more unsure of.  I understand what the sysctl is, but I'm not sure how to come up with a reasonable number.
Generally speaking, and in your experience, would a setting of 2000 achive close to the theoritical meximum of the cards?  Is there an upper limit that I would be worried about?

Random Question:
- is device polling really the answer?  I am missing something in the bge driver that I've overlooked?
- what tunables directly effect processing high volumes of packets.

Network Interfaces:
##################################################################################
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:0b:cd:ca:1d:1a
        inet 209.18.70.211 netmask 0xffffff00 broadcast 209.18.70.255
        inet6 fe80::20b:cdff:feca:1d1a%bge0 prefixlen 64 scopeid 0x1
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:0b:cd:ca:1a:74
        inet 172.16.0.3 netmask 0xfffc0000 broadcast 172.19.255.255
        inet6 fe80::20b:cdff:feca:1a74%bge1 prefixlen 64 scopeid 0x2
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
##################################################################################

I appreciate the help in advance.

Thanks,
Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.