individual queue blocking entire rx unit on ixgbe (Re: How do I balance bandwidth over several virtual NICs?)

[Luigi Rizzo]

reviving this thread:

i am just running experiments on 10.1 beta3 and even
setting dev.ix.*.fc=0 and flipping the interface up and down
does not seem to help: if i read only from a subset of the
queues, the entire rx unit stalls eventually.

I need to drain all queues to keep moving.

Just tested this with 8 instances of netmap-ipfw running
on an 8-core machine (8 queues enabled).

   netmap-ipfw netmap:ix0-0 netmap:ix1-0
   netmap-ipfw netmap:ix0-1 netmap:ix1-1
   ...

and the source on another box is blasting on multiple queues with

   pkt-gen -f tx -i ix0 -d 10.0.10.0-10.0.10.255


I going to look at the driver's code now to see if/how
this issue can be addressed.

cheers
luigi


On Tue, Sep 23, 2014 at 6:00 PM, Adrian Chadd <adrian at freebsd.org> wrote:

> Ah, this behaviour.
>
> It's called DROP_EN on the intel igb / ixgbe hardware. Grep the
> drivers for that particular register bit/setting.
>
> Set that bit for an RX queue and it'll instruct the MAC to drop frames
> destined if that RX ring is full to it and keep receiving on the other
> rings. Otherwise yes, receiving on that ring with the ring full cuases
> the MAC to stop receiving on all rings until that ring has free space.
>
> You flip this on with ixgbe and igb by disabling tx/rx flowcontrol
> (sysctl dev.ix|igb.X.fc=0) before configuring the interface.
>
>
>
> -a
>



-- 
-----------------------------------------+-------------------------------
 Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
 TEL      +39-050-2211611               . via Diotisalvi 2
 Mobile   +39-338-6809875               . 56122 PISA (Italy)
-----------------------------------------+-------------------------------