ixgbe input errors at high data rates

[rihad]

Jack Vogel wrote:
> The 1.3.3 driver is two years old, and your OS is older. I would 
> respectfully suggest
> that you update to 8.0 where lots of effort was put to make 10G hardware 
> perform
> up to its capabilities. Similarly, I have done lots of work in two years 
> to the ixgbe
> driver, I would even suggest that once you have 8 installed you get the 
> driver from
> HEAD.
> 
Thanks, I will certainly make the move to 8.0, but on a new box yet to 
be chosen and ordered. It will take some time before it ships to us. I 
don't think there's a temporary solution, is there?

BTW, slightly off topic: 8.0 seems to be based on libc.so.7, same as 
FreeBSD 7.1. Does this mean I won't be having problems running the 
current ports on 8.0? Unless I run make delete-old-libs, of course :-/


> Regards,
> 
> Jack
> 
> On Thu, Mar 11, 2010 at 7:50 AM, rihad <rihad at mail.ru 
> <mailto:rihad at mail.ru>> wrote:
> 
>     Hi, our Intel 10 GigE cards are finally here, identified as
>     <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.3.3>
>     with the driver ixgbe-1.3.3 off the CD-ROM.
>     One card is used for input, the other for output, doing traffic
>     limiting (dummynet) and accounting in between.
>     At data rates of about 700-1000 mbps netstat -i shows many Input
>     errors on ix0 at a rate of 10-20K per second :(
> 
>     top -HS:
>     CPU:  1.3% user,  0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle
>     Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf,
>     239M Free
>     Swap: 2048M Total, 2048M Free
> 
>      PID USERNAME   PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
>       32 root       -68    -     0K    16K CPU3   3 460:56 100.00%
>     irq258: ix0
>       33 root       -68    -     0K    16K CPU7   7 143:14 100.00% ix0 rxq
>       13 root       171 ki31     0K    16K RUN    5 574:39 93.65% idle: cpu5
>       12 root       171 ki31     0K    16K RUN    6 507:08 88.33% idle: cpu6
>       14 root       171 ki31     0K    16K CPU4   4 424:04 80.37% idle: cpu4
>       18 root       171 ki31     0K    16K CPU0   0 395:34 75.00% idle: cpu0
>       16 root       171 ki31     0K    16K RUN    2 433:10 70.21% idle: cpu2
>      700 root       -68    -     0K    16K -      2 292:19 56.64% dummynet
>       17 root       171 ki31     0K    16K CPU1   1 399:02 50.39% idle: cpu1
>       37 root       -68    -     0K    16K CPU1   1 196:19 39.50% ix1 rxq
>       11 root       171 ki31     0K    16K RUN    7 510:39 14.79% idle: cpu7
>       36 root       -68    -     0K    16K WAIT   5  36:36  8.64%
>     irq260: ix1
>       19 root       -32    -     0K    16K CPU6   6  36:52  5.08% swi4:
>     clock sio
> 
> 
>     Turning dummynet off (by short-circuiting the IPFW rule "allow ip
>     from any to any" before the "pipe tablearg") doesn't eliminate the
>     input errors. Turning ip.fastfowarding off (see below) doesn't help
>     either (why would it), only this time "swi" is chewing up the CPU
>     time instead of "irq". Are we hitting the CPU core limits here? It's
>     a dual cpu quad-core Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (Dell
>     PowerEdge 2950).
>     Shouldn't this $2.5K expensive card have decently-sized hardware
>     buffers to prevent any overruns?
> 
>     Some custom settings:
>     kern.hz=4000
>     net.inet.ip.fastforwarding=1
>     kern.ipc.nmbclusters=111111
>     net.inet.ip.dummynet.io_fast=1
>     net.isr.direct=0
>     net.inet.ip.intr_queue_maxlen=5000
>     hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs
> 
>     FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't
>     used. Should I nonetheless recompile without it? I heard the mere
>     existence of DEVICE_POLLING affects some cards' performance.
> 
>     Thanks for any tips.
>     _______________________________________________
>     freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
>     http://lists.freebsd.org/mailman/listinfo/freebsd-net
>     To unsubscribe, send any mail to
>     "freebsd-net-unsubscribe at freebsd.org
>     <mailto:freebsd-net-unsubscribe at freebsd.org>"
> 
>