individual queue blocking entire rx unit on ixgbe (Re: How do I balance bandwidth over several virtual NICs?)

[Kim Shrier]

Not sure if this is related.  I was testing on 10.1 beta3 and I was copying
approximately 250 GB of data to the test machine, I noticed that the
network would periodically slow down to about 4 mbits/sec or it would
pause for about 20 seconds and then continue at full speed.

The ethernet interface on the test machine is:

bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x57766001> mem 0xa0400000-0xa040ffff,0xa0410000-0xa041ffff irq 16 at device 0.0 on pci1
bge0: CHIP ID 0x57766001; ASIC REV 0x57766; CHIP REV 0x577660; PCI-E


The machine I am transferring from is using the em0 interface:

em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xdc00-0xdc1f mem 0xfb5e0000-0xfb5fffff,0xfb5dc000-0xfb5dffff irq 16 at device 0.0 on pci2


I haven’t noticed any slow down when transferring between machines where
both sides are using the em driver.

During the slow downs and pauses, when I had a top command running,
the process state would show up as “dp->dp” or “rl->l_” instead of something
normal like “select” or CPUn.

If this is related, then maybe the problem is somewhere other than the device
driver.

Kim

On Oct 1, 2014, at 3:48 AM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:

> reviving this thread:
> 
> i am just running experiments on 10.1 beta3 and even
> setting dev.ix.*.fc=0 and flipping the interface up and down
> does not seem to help: if i read only from a subset of the
> queues, the entire rx unit stalls eventually.
> 
> I need to drain all queues to keep moving.
> 
> Just tested this with 8 instances of netmap-ipfw running
> on an 8-core machine (8 queues enabled).
> 
>   netmap-ipfw netmap:ix0-0 netmap:ix1-0
>   netmap-ipfw netmap:ix0-1 netmap:ix1-1
>   ...
> 
> and the source on another box is blasting on multiple queues with
> 
>   pkt-gen -f tx -i ix0 -d 10.0.10.0-10.0.10.255
> 
> 
> I going to look at the driver's code now to see if/how
> this issue can be addressed.
> 
> cheers
> luigi
> 
> 
> On Tue, Sep 23, 2014 at 6:00 PM, Adrian Chadd <adrian at freebsd.org> wrote:
> 
>> Ah, this behaviour.
>> 
>> It's called DROP_EN on the intel igb / ixgbe hardware. Grep the
>> drivers for that particular register bit/setting.
>> 
>> Set that bit for an RX queue and it'll instruct the MAC to drop frames
>> destined if that RX ring is full to it and keep receiving on the other
>> rings. Otherwise yes, receiving on that ring with the ring full cuases
>> the MAC to stop receiving on all rings until that ring has free space.
>> 
>> You flip this on with ixgbe and igb by disabling tx/rx flowcontrol
>> (sysctl dev.ix|igb.X.fc=0) before configuring the interface.
>> 
>> 
>> 
>> -a
>> 
> 
> 
> 
> -- 
> -----------------------------------------+-------------------------------
> Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
> http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
> TEL      +39-050-2211611               . via Diotisalvi 2
> Mobile   +39-338-6809875               . 56122 PISA (Italy)
> -----------------------------------------+-------------------------------
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"