bhyve with vlans - host and vm can't pass traffic

[Scott O'Connell]

I'm very new to bhyve and am having an issue. I'm trying to get VM's and 
VLAN's working.

I'm able to get VLAN's working in a VM, but the VM and the VMHOST, can't 
communicate with each other on the same vlan.

Using 10.1-RELEASE-p9 for both VMHOST01 and DEV. Upstream from the 
VMHOST on lagg0 is a Cisco 3750G.

VMHOST01 before starting VM:

         bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 
0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
           ether f0:1f:af:dd:2e:c5
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect (1000baseT <full-duplex>)
           status: active
         bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 
0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
           ether f0:1f:af:dd:2e:c5
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect (1000baseT <full-duplex>)
           status: active
         lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
           inet6 ::1 prefixlen 128
           inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
           inet 127.0.0.1 netmask 0xff000000
           nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
         lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> 
metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
           ether f0:1f:af:dd:2e:c5
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect
           status: active
           laggproto lacp lagghash l2,l3,l4
           laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
           laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
         vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> 
metric 0 mtu 1500
           options=103<RXCSUM,TXCSUM,TSO4>
           ether f0:1f:af:dd:2e:c5
           inet 10.0.1.17 netmask 0xffffff00 broadcast 10.0.1.255
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect
           status: active
           vlan: 100 parent interface: lagg0

VMHOST after starting VM (added tap0 & bridge0):

         tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 
mtu 1500
           options=80000<LINKSTATE>
           ether 00:bd:70:71:1d:00
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect
           status: no carrier
         bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> 
metric 0 mtu 1500
           ether 02:d3:e4:02:03:00
           nd6 options=1<PERFORMNUD>
           id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
           maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
           root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
           member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 6 priority 128 path cost 2000000
           member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 4 priority 128 path cost 10000


Note that the "status: no carrier" is because I hadn't brought up the VM 
yet. It properly changes to the following after the VM is started:

         tap0: 
flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
           options=80000<LINKSTATE>
           ether 00:bd:70:71:1d:00
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet autoselect
           status: active
           Opened by PID 70827

VM:
         vtnet0: 
flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
           options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
           ether 00:a0:98:2b:34:37
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet 10Gbase-T <full-duplex>
           status: active
         lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
           inet6 ::1 prefixlen 128
           inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
           inet 127.0.0.1 netmask 0xff000000
           nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
         vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> 
metric 0 mtu 1500
           ether 00:a0:98:2b:34:37
           inet 10.0.1.6 netmask 0xffffff00 broadcast 10.0.1.255
           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
           media: Ethernet 10Gbase-T <full-duplex>
           status: active
           vlan: 100 parent interface: vtnet0


With this configuration, both VMHOST01 and DEV can communicate anywhere, 
EXCEPT to each other using their IP on VLAN100.

The ultimate goal is to have more than one VLAN presented to the VM, 
whether it exists on the VMHOST or not.

Where did I go wrong?

Thanks in advance,
scotto