Handling 100.000 packets/sec or more

[David Gilbert]

>>>>> "Adrian" == Adrian Penisoara <ady at freebsd.ady.ro> writes:

Adrian> Hi, At one site that I administer we have a gateway server
Adrian> which services a large SOHO LAN (more than 300 stations) and
Adrian> I'm facing a serious issue: very often we see strong spoofed
Adrian> floods (variable source IP and port, variable destination IP,
Adrian> destination port 80) which can go as far as 100 000
Adrian> packets/sec!

Adrian>   Of course, the server (FreeBSD 5.2-REL, PIII 733Mhz, 256Mb
Adrian> RAM, 3COM 3C905B-TX aka xl0 with checksum offloading support)
Adrian> has a hard time swallowing this kind of traffic. The main
Adrian> issue are the IRQ interrupts: over 15000 interrupts/sec which
Adrian> consume more than 90% of the CPU time.  We got ingress
Adrian> filtering so the packets go no further than the firewall
Adrian> (which, BTW, is not the issue, even disabling it it's the same
Adrian> problem).  The system is still responsive but the load average
Adrian> goes as high as 10 and the interface is losing packets (input
Adrian> errors) which dramatically affects legitimate traffic, besides
Adrian> mbuf(9) starvation. We are taking down the culprit clients,
Adrian> but this takes time and we need the other clients not to be
Adrian> affected by it.

Adrian>   What can I do to make the system better handle this kind of
Adrian> traffic ?  Could device polling(8) or just increasing the
Adrian> kernel frequency clock to 1000Hz or more improve the situation
Adrian> ?  What kind of network cards could face a lot better this
Adrian> burden ? Are there any other solutions ?

Adrian>   On a side note: what would be a adequate formula to
Adrian> calculate the NMBCLUSTERS and MBUFS we should set on this
Adrian> server (via boot-time kern.ipc.nmbclusters and
Adrian> kern.ipc.nmbufs) ?

In our experience, switch to fxp ethernet cards, test several
motherboards and enable polling.

fxp and em cards appear to have the best performance ... outrunning
other cards by a fair margin.

Different motherboards have several orders of magnitude different
performance with the same processor.

Polling (as others have mentioned) roughly doubles the throughput of a
server  and eliminates live lock.

Dave.

-- 
============================================================================
|David Gilbert, Independent Contractor.       | Two things can only be     |
|Mail:       dave at daveg.ca                    |  equal if and only if they |
|http://daveg.ca                              |   are precisely opposite.  |
=========================================================GLO================