Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

Dual opteron 270
32 bit GENERIC KERNEL Nothing changed in sysctl except forwarding and ip 
forwarding
Broadcom interfaces on board NIC

last pid: 11557;  load averages:  1.13,  0.83,  
0.48                                                                  up 
0+03:24:26  21:58:38
70 processes:  6 running, 46 sleeping, 18 waiting
CPU states:  0.0% user,  0.0% nice,  0.2% system, 28.8% interrupt, 71.1% 
idle
Mem: 9124K Active, 6844K Inact, 26M Wired, 9776K Buf, 1957M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   13 root     171 ki31     0K     8K CPU1   1 203:39 98.93% idle: cpu1
   14 root     171 ki31     0K     8K RUN    0 203:26 98.05% idle: cpu0
   33 root     -68    -     0K     8K CPU2   2  13:06 93.99% irq24: bge0
   11 root     171 ki31     0K     8K CPU3   3 201:59 79.10% idle: cpu3
   34 root     -68    -     0K     8K WAIT   3   1:49 18.90% irq25: bge1
   12 root     171 ki31     0K     8K RUN    2 190:42  4.00% idle: cpu2

           input         (bge0)           output
   packets  errs      bytes    packets  errs      bytes colls
    410143 65619   24608586          1     0        226     0
    410350 66749   24621006          1     0        178     0
    409452 65382   24567126          1     0        178     0
    410338 63157   24620286          1     0        178     0
    410125 65777   24607446          1     0        178     0
    409794 63018   24587706          1     0        178     0
    408208 67566   24492486          1     0        178     0
    408416 70305   24504906          1     0        178     0
    407919 68339   24475206          1     0        178     0


sysctl -a | grep bge.0.stats
dev.bge.0.stats.FramesDroppedDueToFilters: 0
dev.bge.0.stats.DmaWriteQueueFull: 310574781
dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
dev.bge.0.stats.NoMoreRxBDs: 0
dev.bge.0.stats.InputDiscards: 20942213
dev.bge.0.stats.InputErrors: 15
dev.bge.0.stats.RecvThresholdHit: 51440366
dev.bge.0.stats.DmaReadQueueFull: 0
dev.bge.0.stats.DmaReadHighPriQueueFull: 0
dev.bge.0.stats.SendDataCompQueueFull: 0
dev.bge.0.stats.RingSetSendProdIndex: 25738223
dev.bge.0.stats.RingStatusUpdate: 51719806
dev.bge.0.stats.Interrupts: 51719806
dev.bge.0.stats.AvoidedInterrupts: 0
dev.bge.0.stats.SendThresholdHit: 0
dev.bge.0.stats.rx.Octets: 3477501149
dev.bge.0.stats.rx.Fragments: 116
dev.bge.0.stats.rx.UcastPkts: 519429709
dev.bge.0.stats.rx.MulticastPkts: 0
dev.bge.0.stats.rx.FCSErrors: 3
dev.bge.0.stats.rx.AlignmentErrors: 0
dev.bge.0.stats.rx.xonPauseFramesReceived: 0
dev.bge.0.stats.rx.xoffPauseFramesReceived: 0
dev.bge.0.stats.rx.ControlFramesReceived: 0
dev.bge.0.stats.rx.xoffStateEntered: 0
dev.bge.0.stats.rx.FramesTooLong: 0
dev.bge.0.stats.rx.Jabbers: 0
dev.bge.0.stats.rx.UndersizePkts: 0
dev.bge.0.stats.rx.inRangeLengthError: 0
dev.bge.0.stats.rx.outRangeLengthError: 0
dev.bge.0.stats.tx.Octets: 2215096864
dev.bge.0.stats.tx.Collisions: 0
dev.bge.0.stats.tx.XonSent: 0
dev.bge.0.stats.tx.XoffSent: 0
dev.bge.0.stats.tx.flowControlDone: 0
dev.bge.0.stats.tx.InternalMacTransmitErrors: 0
dev.bge.0.stats.tx.SingleCollisionFrames: 0
dev.bge.0.stats.tx.MultipleCollisionFrames: 0
dev.bge.0.stats.tx.DeferredTransmissions: 0
dev.bge.0.stats.tx.ExcessiveCollisions: 0
dev.bge.0.stats.tx.LateCollisions: 0
dev.bge.0.stats.tx.UcastPkts: 25738364
dev.bge.0.stats.tx.MulticastPkts: 0
dev.bge.0.stats.tx.BroadcastPkts: 15
dev.bge.0.stats.tx.CarrierSenseErrors: 0
dev.bge.0.stats.tx.Discards: 0
dev.bge.0.stats.tx.Errors: 0

errors..ERRORS!)(@!*()

Hitting that 400k pps limit again, and this is a slower machine than my 
dual 2212.
Going to compile 7-STABLE with options for the cpu and will report back