ixgbe input errors at high data rates

[rihad]

Hi, our Intel 10 GigE cards are finally here, identified as
<Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.3.3>
with the driver ixgbe-1.3.3 off the CD-ROM.
One card is used for input, the other for output, doing traffic limiting 
(dummynet) and accounting in between.
At data rates of about 700-1000 mbps netstat -i shows many Input errors 
on ix0 at a rate of 10-20K per second :(

top -HS:
CPU:  1.3% user,  0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle
Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf, 239M Free
Swap: 2048M Total, 2048M Free

   PID USERNAME   PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
    32 root       -68    -     0K    16K CPU3   3 460:56 100.00% irq258: ix0
    33 root       -68    -     0K    16K CPU7   7 143:14 100.00% ix0 rxq
    13 root       171 ki31     0K    16K RUN    5 574:39 93.65% idle: cpu5
    12 root       171 ki31     0K    16K RUN    6 507:08 88.33% idle: cpu6
    14 root       171 ki31     0K    16K CPU4   4 424:04 80.37% idle: cpu4
    18 root       171 ki31     0K    16K CPU0   0 395:34 75.00% idle: cpu0
    16 root       171 ki31     0K    16K RUN    2 433:10 70.21% idle: cpu2
   700 root       -68    -     0K    16K -      2 292:19 56.64% dummynet
    17 root       171 ki31     0K    16K CPU1   1 399:02 50.39% idle: cpu1
    37 root       -68    -     0K    16K CPU1   1 196:19 39.50% ix1 rxq
    11 root       171 ki31     0K    16K RUN    7 510:39 14.79% idle: cpu7
    36 root       -68    -     0K    16K WAIT   5  36:36  8.64% irq260: ix1
    19 root       -32    -     0K    16K CPU6   6  36:52  5.08% swi4: 
clock sio


Turning dummynet off (by short-circuiting the IPFW rule "allow ip from 
any to any" before the "pipe tablearg") doesn't eliminate the input 
errors. Turning ip.fastfowarding off (see below) doesn't help either 
(why would it), only this time "swi" is chewing up the CPU time instead 
of "irq". Are we hitting the CPU core limits here? It's a dual cpu 
quad-core Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (Dell PowerEdge 2950).
Shouldn't this $2.5K expensive card have decently-sized hardware buffers 
to prevent any overruns?

Some custom settings:
kern.hz=4000
net.inet.ip.fastforwarding=1
kern.ipc.nmbclusters=111111
net.inet.ip.dummynet.io_fast=1
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000
hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs

FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't used. 
Should I nonetheless recompile without it? I heard the mere existence of 
DEVICE_POLLING affects some cards' performance.

Thanks for any tips.