pf performance?

[Erich Weiler]

Hello all,

I have a question here about how FreeBSD (8.1-RELEASE-p13 specifically) 
behaves when acting as a firewall.  I understand the pf process is 
"giant locked" to a single CPU core when inspecting packets inbound and 
outbound.  I was wondering, how does that manifest when I look at "top 
-P" on the firewall?

Right now I have a dual port Myricom 10G NIC (packets inbound on one 
interface and outbound on the other), and the mxge driver is 
"multiplexing" interrupt processing across all the CPU cores for speed. 
  So, when the firewall is busy, I see all the cpu cores quite busy 
processing interrupts (like 70% or more CPU utilization).  But, all CPU 
work seems to be in interrupts.  I don't see anything, or *very* little, 
in system or user space for CPU utilization.  Should the pf process be 
using some CPU too?  If so, how could I tell that?  I'm trying to figure 
out if I'm limited by not having enough CPU to process the interrupts or 
not enough CPU to process the packet filtering process.  Right now it 
looks like interrupts but I'm not sure.

The Myricom folks looked at our debugging info on the mxge driver and 
say that based on what they see, mxge is dropping packets because the 
host cannot pull packets out of the NIC buffer fast enough.  The host is 
using a four core Xeon X5677 3.46GHz CPU.  We're processing 140,000 
packets per second or so, and I see rates up to several gigabits per 
second, but all my research seems to indicate it can do better than 
that, and that we should not be dropping packets.  Or maybe the question 
is: why doesn't the host pull the packets from the NIC fast enough?  Is 
the CPU tied up doing something else?  Interrupts?

Does anyone have any ideas?  TIA!!

Thanks!
-erich