Re: issue while bridging VF on mellanox
- Reply: Benoit Chesneau : "Re : Re: issue while bridging VF on mellanox"
- In reply to: benoitc@enki-multimedia.eu: "Re: issue while bridging VF on mellanox"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 17 Mar 2025 18:25:02 UTC
Hey, haven’t tested with SR-IOV but can test it later if you want. Santi > On 17 Mar 2025, at 17:50, benoitc@enki-multimedia.eu wrote: > > Oh, i need to check if the vxlan works correctly indeed... > >> So you are also using sr-iov ? > > Benoît > >> On 17 Mar 2025, at 17:08, Santiago Martinez <sm@codenetworks.net> wrote: >> >> Hi Benoit, one more thing, Im not sure whats the current support for v6, specially for ND. >> >> Best regards. >> >> Santi >> >> >> >> On 3/17/25 16:48, Santiago Martinez wrote: >>> Hi Benoit, >>> >>> I have a similar setup to yours (not using MLX4), and it works. >>> >>> I have two VMs acting as routers (R-A and R-B), each with two interfaces. >>> >>> On Router-A: >>> >>> VTNET1 → P2P to Router-B. >>> VTNET0 → Sub-interface (VLAN 333) - Client-A. >>> VXLAN0 → Towards Router-B. >>> BR-TEST → VXLAN0 and VTNET0.333 as members. >>> On Router-B: >>> >>> VTNET1 → P2P to Router-A. >>> VTNET0 → Untagged - Client-B. >>> VXLAN0 → Towards Router-A. >>> BR-TEST → VXLAN0 and VTNET0 as members. >>> Traffic passes. >>> >>> One thing to note—VXLAN0 will appear as UP/Active even if it’s not reaching the other side. >>> >>> Best regards, >>> Santi >>> >>> >>> >>> On 3/17/25 15:32, Benoit Chesneau wrote: >>>> Hi all, >>>> >>>> I'm trying to make some kind of cross connect between a vlan and a vxlan, but as far as I understand the mac address are not learnt from the vlan When I listen using tcpdump on the vlan i can see the arp request but not in the in the vxlan. ANy idea how should I fix this? Is there any other way to do a cross conect or shoud I go back to not use sriov when i need to build bridges? >>>> >>>> >>>> Any idea is welcome. >>>> >>>> Benoit >>>> >>>> More details follow >>>> >>>> The vlan is created over a VF device in a bhyve vm: >>>> >>>> vlan601: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1356 >>>> options=18080000<LINKSTATE,TXTLS4,TXTLS6> >>>> ether 4a:75:f1:f5:a5:0a >>>> inet6 fe80::4875:f1ff:fef5:a50a%vlan601 prefixlen 64 scopeid 0x5 >>>> groups: vlan >>>> vlan: 601 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 >>>> media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause> >>>> status: active >>>> nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> >>>> >>>> The vxlan is >>>> >>>> vxlan0: flags=1008903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1356 >>>> options=80020<JUMBO_MTU,LINKSTATE> >>>> ether 58:9c:fc:10:12:01 >>>> inet6 fe80::5a9c:fcff:fe10:1201%vxlan0 prefixlen 64 tentative scopeid 0x7 >>>> groups: vxlan >>>> vxlan vni 400 local [2001:db8:1000::1]:4789 remote [2001:db8:1000::2]:4789 >>>> media: Ethernet autoselect (autoselect <full-duplex>) >>>> status: active >>>> nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> >>>> >>>> >>>> And the bridge: >>>> >>>> # ifconfig bridge0 >>>> bridge0: flags=1028943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC,LOWER_UP> metric 0 mtu 1356 >>>> options=0 >>>> ether 58:9c:fc:10:ff:d3 >>>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>>> member: vlan601 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> >>>> ifmaxaddr 0 port 5 priority 128 path cost 800 >>>> member: vxlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> >>>> ifmaxaddr 0 port 7 priority 128 path cost 55 >>>> groups: bridge >>>> nd6 options=3<PERFORMNUD,ACCEPT_RTADV> >>>> >>>> The PF on the machine is a Mellanox [ConnectX-4 Lx >>>> >>>> # pciconf -lvbce mlx5_core0 >>>> mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 >>>> vendor = 'Mellanox Technologies' >>>> device = 'MT27710 Family [ConnectX-4 Lx]' >>>> class = network >>>> subclass = ethernet >>>> bar [10] = type Prefetchable Memory, range 64, base 0xe4000000, size 33554432, enabled >>>> cap 10[60] = PCI-Express 2 endpoint max data 256(512) FLR RO NS >>>> max read 4096 >>>> link x8(x8) speed 8.0(8.0) ASPM L1(L1) >>>> cap 03[48] = VPD >>>> cap 11[9c] = MSI-X supports 64 messages, enabled >>>> Table in map 0x10[0x2000], PBA in map 0x10[0x3000] >>>> cap 09[c0] = vendor (length 24) >>>> cap 01[40] = powerspec 3 supports D0 D3 current D0 >>>> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected >>>> ecap 000e[150] = ARI 1 >>>> ecap 0010[180] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled >>>> 8 VFs configured out of 16 supported >>>> First VF RID Offset 0x0002, VF RID Stride 0x0001 >>>> VF Device ID 0x1016 >>>> Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 2097152, 4194304 >>>> iov bar [1a4] = type Prefetchable Memory, range 64, base 0xc7ffe000000, size 1048576, enabled >>>> ecap 0019[1c0] = PCIe Sec 1 lane errors 0 >>>> ecap 000d[230] = ACS 1 Source Validation unavailable, Translation Blocking unavailable >>>> P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable >>>> P2P Upstream Forwarding unavailable, P2P Egress Control unavailable >>>> P2P Direct Translated unavailable, Enhanced Capability unavailable >>>> PCI-e errors = Correctable Error Detected >>>> Unsupported Request Detected >>>> Corrected = Advisory Non-Fatal Error >>>> >>>> Other side is the same except that the card is an intel one: >>>> >>>> # pciconf -lvbce ixl0 >>>> ixl0@pci0:23:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1572 subvendor=0x15d9 subdevice=0x093b >>>> vendor = 'Intel Corporation' >>>> device = 'Ethernet Controller X710 for 10GbE SFP+' >>>> class = network >>>> subclass = ethernet >>>> bar [10] = type Prefetchable Memory, range 64, base 0xc4800000, size 8388608, enabled >>>> bar [1c] = type Prefetchable Memory, range 64, base 0xc5808000, size 32768, enabled >>>> cap 01[40] = powerspec 3 supports D0 D3 current D0 >>>> cap 05[50] = MSI supports 1 message, 64 bit, vector masks >>>> cap 11[70] = MSI-X supports 129 messages, enabled >>>> Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] >>>> cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR RO >>>> max read 512 >>>> link x8(x8) speed 8.0(8.0) >>>> cap 03[e0] = VPD >>>> ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected >>>> ecap 0003[140] = Serial 1 b8f678ffff55c27c >>>> ecap 000e[150] = ARI 1 >>>> ecap 0010[160] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled >>>> 16 VFs configured out of 64 supported >>>> First VF RID Offset 0x0010, VF RID Stride 0x0001 >>>> VF Device ID 0x154c >>>> Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 >>>> iov bar [184] = type Prefetchable Memory, range 64, base 0xc5000000, size 65536, enabled >>>> iov bar [190] = type Prefetchable Memory, range 64, base 0xc5100000, size 16384, enabled >>>> ecap 0017[1a0] = TPH Requester 1 >>>> ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable >>>> P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable >>>> P2P Upstream Forwarding unavailable, P2P Egress Control unavailable >>>> P2P Direct Translated unavailable, Enhanced Capability unavailable >>>> ecap 0019[1d0] = PCIe Sec 1 lane errors 0 >>>> PCI-e errors = Correctable Error Detected >>>> Unsupported Request Detected >>>> Corrected = Advisory Non-Fatal Error >