ixgbe input errors at high data rates

[Jack Vogel]

The 1.3.3 driver is two years old, and your OS is older. I would
respectfully suggest
that you update to 8.0 where lots of effort was put to make 10G hardware
perform
up to its capabilities. Similarly, I have done lots of work in two years to
the ixgbe
driver, I would even suggest that once you have 8 installed you get the
driver from
HEAD.

Regards,

Jack

On Thu, Mar 11, 2010 at 7:50 AM, rihad <rihad at mail.ru> wrote:

> Hi, our Intel 10 GigE cards are finally here, identified as
> <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.3.3>
> with the driver ixgbe-1.3.3 off the CD-ROM.
> One card is used for input, the other for output, doing traffic limiting
> (dummynet) and accounting in between.
> At data rates of about 700-1000 mbps netstat -i shows many Input errors on
> ix0 at a rate of 10-20K per second :(
>
> top -HS:
> CPU:  1.3% user,  0.0% nice, 25.2% system, 14.1% interrupt, 59.3% idle
> Mem: 1047M Active, 2058M Inact, 466M Wired, 126M Cache, 214M Buf, 239M Free
> Swap: 2048M Total, 2048M Free
>
>  PID USERNAME   PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
>   32 root       -68    -     0K    16K CPU3   3 460:56 100.00% irq258: ix0
>   33 root       -68    -     0K    16K CPU7   7 143:14 100.00% ix0 rxq
>   13 root       171 ki31     0K    16K RUN    5 574:39 93.65% idle: cpu5
>   12 root       171 ki31     0K    16K RUN    6 507:08 88.33% idle: cpu6
>   14 root       171 ki31     0K    16K CPU4   4 424:04 80.37% idle: cpu4
>   18 root       171 ki31     0K    16K CPU0   0 395:34 75.00% idle: cpu0
>   16 root       171 ki31     0K    16K RUN    2 433:10 70.21% idle: cpu2
>  700 root       -68    -     0K    16K -      2 292:19 56.64% dummynet
>   17 root       171 ki31     0K    16K CPU1   1 399:02 50.39% idle: cpu1
>   37 root       -68    -     0K    16K CPU1   1 196:19 39.50% ix1 rxq
>   11 root       171 ki31     0K    16K RUN    7 510:39 14.79% idle: cpu7
>   36 root       -68    -     0K    16K WAIT   5  36:36  8.64% irq260: ix1
>   19 root       -32    -     0K    16K CPU6   6  36:52  5.08% swi4: clock
> sio
>
>
> Turning dummynet off (by short-circuiting the IPFW rule "allow ip from any
> to any" before the "pipe tablearg") doesn't eliminate the input errors.
> Turning ip.fastfowarding off (see below) doesn't help either (why would it),
> only this time "swi" is chewing up the CPU time instead of "irq". Are we
> hitting the CPU core limits here? It's a dual cpu quad-core Intel(R) Xeon(R)
> CPU E5410 @ 2.33GHz (Dell PowerEdge 2950).
> Shouldn't this $2.5K expensive card have decently-sized hardware buffers to
> prevent any overruns?
>
> Some custom settings:
> kern.hz=4000
> net.inet.ip.fastforwarding=1
> kern.ipc.nmbclusters=111111
> net.inet.ip.dummynet.io_fast=1
> net.isr.direct=0
> net.inet.ip.intr_queue_maxlen=5000
> hw.intr_storm_threshold=8000 #as suggested by the ixgbe-1.3.3 docs
>
> FreeBSD 7.1 kernel built with DEVICE_POLLING, even though it isn't used.
> Should I nonetheless recompile without it? I heard the mere existence of
> DEVICE_POLLING affects some cards' performance.
>
> Thanks for any tips.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>