Network performance in a dual CPU system

[dima]

> Hello all,
> 
> thanks for the replies. Most of you have suggested that I turn on 
> polling and give it a try. The machine is in production, hence I need 
> to schedule downtime for that.
> 
> The system is mainly being used as a dedicated router. It runs OSPF, 
> BGP and IPFW (around 150 rules). OSPF and BGP are managed by Quagga. 
> The box has 2 gigabit interfaces that handle on average 200Mbp/s - 50K 
> packets/s (inbound and outbound combined), each one of them.

The second CPU wouldn't help you for sure. There's only one [swi1: net] kernel thread which deals with all the kernel traffic. The option of per-CPU [swi: net] threads was discussed on freebsd-arch@ several months ago, but it wouldn't be implemented soon. So, the only hardware option is installing the fastest CPU possible.
There are several software (FreeBSD specific) options though:
1. You should surely try polling(4). 50kpps mean 50000 interrupts and the same amount of context switches, which are quite expensive.
2. FastForwarding. It's the most suitable for you. As I know, Quagga inserts its dynamic routes to the system routing table. And FastForwarding is aware of routing table and firewall rules. And the most exciting: you can switch it on/off without reboot:
# sysctl net.inet.ip.fastforwarding=1
The only limitation is it applies to IPv4 unicast traffic only. There's no documentation on this feature as i know (am I wrong, or should I report this as a documentation bug?) but you can look at the comments in the beginning of /sys/netinet/ip_fastfwd.c
The authors reported up to 1Mpps (see page 10 at http://people.freebsd.org/~andre/FreeBSD-5.3-Networking.pdf)

> 
> 
> Some of you have asked for the following information:
> 
> 
> - As I indicated before, polling is currently disabled.
> 
> 
> - Hyperthreading (HTT) is disabled.
> 
> 
> mull [~]$vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                        3466          0
> irq6: fdc0                            10          0
> irq13: npx0                            1          0
> irq14: ata0                           47          0
> irq21: fxp1                     20462527          8
> irq28: bge0                   3511765157       1444
> irq29: bge1                   3633124373       1494
> irq30: aac0                      1842472          0
> cpu0: timer                    566751007        233
> Total                         7733949060       3181
> 
> 
> mull [~]$netstat -m
> 644/646/1290 mbufs in use (current/cache/total)
> 643/407/1050/17088 mbuf clusters in use (current/cache/total/max)
> 0/5/4528 sfbufs in use (current/peak/max)
> 1447K/975K/2422K bytes allocated to network (current/cache/total)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
> 
> 
> 
> Thank you,
> 
> --
> Marcos