bridge interface IP connectivity issue when using oce interface

Tue Aug 22 08:39:35 UTC 2017

Ok thanks Eugene.
net.link.bridge.inherit_mac=1 helped get the connectivity from the bridge
however, when I start a FreeBSD bhyve VM and attached that to a tap
interface in the bridge, I don't get connectivity from the VM.

SETUP:
Gateway - 192.168.0.222/29
Server - 192.168.0.218/29
VM - 192.168.0.219/29

On the VM, I see the ARP entries for the GW and the VM itself but cannot
ping the gateway nor the host.

--------- on the VM --------------
# uname -a
FreeBSD  11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28
UTC 2017     root at releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64
# ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:52:c8:33
hwaddr 00:a0:98:52:c8:33
inet 192.168.0.219 netmask 0xfffffff8 broadcast 192.168.0.223
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active

# ping -c4 192.168.0.222
PING 192.168.0.222 (192.168.0.222): 56 data bytes
^C
--- 192.168.0.222 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

# arp -an
? (192.168.0.219) at 00:a0:98:52:c8:33 on vtnet0 permanent [ethernet]
? (192.168.0.218) at (incomplete) on vtnet0 expired [ethernet]
? (192.168.0.222) at 00:08:e3:ff:fd:90 on vtnet0 expires in 1126 seconds
[ethernet]

----------- end of VM ----------------

----------- on the host ---------------
root at bsdcan:~ # uname -a
FreeBSD bsdcan 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21
02:08:28 UTC 2017     root at releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64
root at bsdcan:~ # ifconfig
[..]
oce3: flags=8143<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1500
options=500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO>
ether 90:1b:0e:98:d3:93
hwaddr 90:1b:0e:98:d3:93
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
[..]
tap0: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=80000<LINKSTATE>
ether 00:bd:0f:bb:27:00
hwaddr 00:bd:0f:bb:27:00
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 81874
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
ether 90:1b:0e:98:d3:93
inet 192.168.0.218 netmask 0xfffffff8 broadcast 192.168.0.223
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 7 priority 128 path cost 55
member: oce3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
       ifmaxaddr 0 port 4 priority 128 path cost 2000
root at bsdcan:~ # ifconfig bridge0 addr
00:08:e3:ff:fd:90 Vlan1 oce3 1200 flags=0<>

root at bsdcan:~ # ps aux | grep vmrun
root  47167    0.0  0.0   14828  2396  1  S+   04:08       0:00.00 grep
vmrun
root  73264    0.0  0.0   13180  2740  2  I+   03:39       0:00.00 sh
/usr/share/examples/bhyve/vmrun.sh -c 10 -m 8192M -t tap0 -d guest.img -i
-I FreeBSD-11.1-RELEASE-amd

root at bsdcan:~ # arp -an
? (192.168.0.218) at 90:1b:0e:98:d3:93 on bridge0 permanent [bridge]
? (192.168.0.222) at 00:08:e3:ff:fd:90 on bridge0 expires in 1191 seconds
[bridge]

root at bsdcan:~ # sysctl net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 1
net.link.bridge.inherit_mac: 1
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 1
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 1
net.link.bridge.pfil_onlyip: 1
--------------- end of host ----------

Shouldn't the VM mac address show up in the MAC address table of the
bridge0 ?
When I 'tcpdump -i tap0 -vv' I see literally only the ARP request from the
.222 towards the VM and nothing back from the VM at all which does not make
a lot of sense since I get the '-t tap0' when launching the VM, I would
expect some traffic on the tap0 intf from the VM.

Any thoughts on where filtering could happen?

I assume the VM should be able to ping the IP set on the bridge0. Is it
fair assumption?

Thanks.

On Mon, Aug 21, 2017 at 9:14 PM, Eugene Grosbein <eugen at grosbein.net> wrote:

> 22.08.2017 7:49, Boris пишет:
> > Hi all,
> >
> > I have two environments.
> >
> > Environment A:
> > Server running fresh install of 11.1-RELEASE with bge physical NIC.
> > If I just configure a bridge interface, add a physical NIC which has
> > working connectivity, say bge3, and add an IP address on the bridge
> > interface in the same subnet as bge3, I can ping that IP from any host on
> > the LAN.
> >
> > Environment B:
> > Server running fresh install of 11.1-RELEASE with oce physical NIC.
> > If I just configure a bridge interface, add a physical NIC which has
> > working connectivity, say oce3, and add an IP address on the bridge
> > interface in the same subnet as oce3, I CANNOT ping that IP from anywhere
> > on the LAN.
>
> First, when you add member interfaces to a bridge, you should move all
> their
> IP addresses to the bridge. That is, bridge member interfaces should
> have no IP addresses, only bridge itself.
>
> Second, you should re-read bridge(4) manual page and use
> sysctl net.link.bridge.inherit_mac=1 and use physical NIC as first
> bridge member so that your uplink has no reasons to filter
> traffic of the bridge due to its fabricated MAC.
>
>
> >
> > I need the bridge as I would like to have bhyve VM's connected through
> that
> > bridge to the outside - plain bridged networking, no NAT or anything
> else.
> > Unfortunately, the VM does not have any connectivity to the outside. What
> > is weird is that I see the ARP entries in the VM for its gateway, I see
> the
> > MAC addresses in the bridge for the VM and the gateway, but no IP
> > connectivity seems to work - ping fails.
> > I disabled TX checksum and other things using 'ifconfig oce3 -txcsum -lro
> > -tso' to avoid messages around capabilities issues when adding a tap
> > interface which does not have the same features as the physical
> interface.
> > So far, I have not been able to get IP connectivity to the VM.
> >
> > In terms of documentation, I have used the handbook to create the VM:
> > https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
> >
> > I have used the handbook to create the bridge:
> > https://www.freebsd.org/doc/handbook/network-bridging.html
> >
> > Under 30.6.1, it says, I should be able to configure the bridge with an
> IP
> > address which seems to fail when the 'oce' interface is used.
> >
> > Would anybody have any pointer at what to do next to help identify the
> > issue?
> >
> > Thanks !
> > _______________________________________________
> > freebsd-net at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> >
> >
>
>