Netmap bridge not working with 10G Ethernet ports

Tue Nov 24 02:57:33 UTC 2020

Hi Vincenzo,

Thanks for pointing this.

On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione <vmaffione at freebsd.org>
wrote:

> # ifconfig ix0 promisc
> # ifconfig ix1 promisc
>
> This is an additional requirement when using netmap bridge, because that
> is not done automatically (differently from what happens with if_bridge(4)).
> If promisc is not enabled, the NIC will drop any unicast packet that is
> not directed to the NIC's address (e.g. the ARP reply in your case).
> Broadcast packets will of course pass (e.g. the ARP request). This explains
> the absence of IRQs and the head/tail pointers not being updated.
> So no bugs AFAIK.
>

Setting the interfaces in promiscous mode makes things to work properly.

I tried the same with AMD Ports and it's still not working.  I believe this
is something specific to if_axp driver. I will see what is going wrong with
packet forwarding with AMD ports. Thanks for pointing this out.

I figured it out the hard way, but it was actually also documented on the
> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
> I will add it to the netmap bridge man page.
>

That would be helpful. Thanks.

> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
> vmaffione at freebsd.org> ha scritto:
>
>>
>>
>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <rajfbsd at gmail.com>
>> ha scritto:
>>
>>> Hi Vincenzo,
>>>
>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <vmaffione at freebsd.org>
>>> wrote:
>>>
>>>>
>>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4)
>>>> uses iflib(4). This means that actually if_axp(4) has native netmap
>>>> support, because iflib(4) has native netmap support.
>>>>
>>>>
>>> It means that the driver has some modifications to allow netmap to
>>>> directly program the NIC rings. These modifications are mostly the
>>>> per-driver txsync and rxsyng routines.
>>>> In case of iflib(4) drivers, these modifications are provided directly
>>>> within the iflib(4) code, and therefore any driver using iflib will have
>>>> native netmap support.
>>>>
>>>
>>> Thanks for clarifying on the Native Netmap support.
>>>
>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
>>>> are basically hitting the same issue of if_axp(4)
>>>> At this point I must think that there is still some issue with the
>>>> interaction between iflib(4) and netmap(4).
>>>>
>>>
>>> Ok. Let me know if any more debug info needed in this part.
>>>
>>> I see. This info may be useful. Have you tried to look at interrupts
>>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
>>>> ARP replies)?
>>>>
>>>
>>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>>> any ARP reply packet getting dumped.
>>>
>>>
>>> *irq59: ix0:rxq0                        1          0 (only 1 interrupt
>>> on the opposite side)*irq67: ix0:aq                          2
>>>  0
>>>
>>> *irq68: ix1:rxq0                        3          0  (you can see 3
>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>>              2          0
>>>
>>> The same experiment, when I try with AMD inbuilt ports, I don't see that
>>> 1 interrupt also raised.
>>>
>>> irq81: ax0:dev_irq                    16          0
>>> irq83: ax0                          2541          4
>>> irq93: ax1:dev_irq                    27          0
>>> irq95: ax1                          2371          3
>>> *irq97: ax1:rxq0                        3          0 (you can see 3
>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>>> "ax0:rxq0" for ARP reply from System 2)*
>>>
>>> I will do some more testing to see whether this behavior is consistent
>>> or intermittent.
>>>
>>> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>>>> the same as ix(4) and if_axp(4).
>>>> This is something that I'm not able to understand right now.
>>>> It does not look like something related to offloads.
>>>>
>>>> Next week I will try to see if I can reproduce your issue with em(4),
>>>> and report back. That's still an Intel driver using iflib(4).
>>>>
>>>
>>> The "igb(4)" driver, with which things are working now is related to
>>> em(4) driver (may be for newer hardware version).  Initially we faced
>>> similar issue with igb(4) driver as well. After reverting the following
>>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>>
>>>
>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>>
>>> Thanks for providing your inputs on this issue Vincenzo.  Let me know
>>> for any more details that you need.
>>>
>>>
>> I was able to reproduce your issue on FreeBSD-CURRENT running within a
>> QEMU VM, with two em(4) devices and the netmap bridge running between them.
>> I see the ARP request packet received on em0 (with associated IRQ), and
>> forwarded on em1. However, the ARP reply coming on em1 does not trigger an
>> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
>> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
>> and lets me think that the issue is more likely driver-related than
>> netmap/iflib-related.
>> In any case, would you mind filing the issue on the bugzilla, so that we
>> can properly track this issue?
>>
>> Thanks,
>>   Vincenzo
>>
>>
>>> Thanks,
>>> Rajesh.
>>>
>>