ipv4 routing from bhyve

[Willem Jan Withagen]

On 11-1-2015 22:32, williamecowell at hush.ai wrote:
> Hello, I hope I can have some assistance.
> 
> I am trying to get networking via wlan0 but without NAT or bridging (doesn't work on wifi unless WDS).
> 
> say my my main network is 10.10.2.0/24, gateway/internet is 10.10.2.1, my ip is 10.10.2.252.
> 
> I started to config my bhyve network on 172.16.32.0/24
> 
> I added a bridge interface with an ip of 172.16.32.1
> 
> enable forwarding and fastforwarding. from my understanding of the handbook chapter things should work when I type:
> 
> # route add -net 172.16.32.0/24 10.10.2.252
> route: writing to routing socket: File exists
> add net 172.16.32.0: gateway 10.10.2.252 fib 0: route already in table
> # 
> 
> # netstat -4nr
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags      Netif Expire
> default            10.10.2.1        UGS       lagg0
> 127.0.0.1          link#3             UH          lo0
> 10.10.2.0/24     link#5             U         lagg0
> 10.10.2.252      link#5             UHS         lo0
> 172.16.32.0/24    link#4             U       bridge0
> 172.16.32.1       link#4             UHS         lo0
> # 
> 
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	ether 00:bd:0f:fc:01:10
> 	inet 172.16.32.1 netmask 0xffffff00 broadcast 172.16.32.255 
> 	nd6 options=9<PERFORMNUD,IFDISABLED>
> 	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> 	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> 	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> 	member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> 	        ifmaxaddr 0 port 6 priority 128 path cost 2000000
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> ...
> 	inet 

 netmask 0xffffff00 broadcast 10.10.2.255
> 	nd6 options=9<PERFORMNUD,IFDISABLED>
> 	media: Ethernet autoselect
> 	status: active
> 	laggproto failover lagghash l2,l3,l4
> 	laggport: alc0 flags=1<MASTER>
> 	laggport: wlan0 flags=4<ACTIVE>
> tap0: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=80000<LINKSTATE>
> 	ether 00:bd:8f:62:67:10
> 	nd6 options=9<PERFORMNUD,IFDISABLED>
> 	media: Ethernet autoselect
> 	status: no carrier
> wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> ...
> pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
> tap9: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=80000<LINKSTATE>
> 	ether 00:bd:cb:46:02:09
> 	nd6 options=1<PERFORMNUD>
> 	media: Ethernet autoselect
> 	status: no carrier
> tap1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=80000<LINKSTATE>
> 	ether 00:bd:58:61:02:01
> 	nd6 options=1<PERFORMNUD>
> 	media: Ethernet autoselect
> 	status: no carrier

Well one of the things of concern is the fact that your tap interfaces
have:
	status: no carrier

My connected bhyve vm's have, amongst others:
	status: active
        groups: tap
        Opened by PID 20763

And my bridge device tells me:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
        ether 02:76:2d:3d:9c:00
        inet xxx.xxx.xxx.xxx netmask 0xff000000 broadcast 37.255.255.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap651 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 55
        member: tap14041 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: tap13101 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: tap12041 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
        member: tap13 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000


So I think you first need to connect your VM's, before anything else
will start to work. Like adding the tap-ifs to the bridge.

And on the host itself you don't really need to add routing for the VM's
because everything is actually already connected. Which is what the
netstat output tells you. The routing table tells you that traffic for
	172.16.32.0/24    link#4             U       bridge0
is send into the the bridge0 devices, which is directly connected.
And ip-nrs in that range should appear in the the arp table.
And the host then knows how to get to them directly.

Routing for 172.16.32.0/24, if any needed, will be required on other
hosts on you network on lagg0. Unless all hosts there have 10.10.2.252
as their default route.

Regards,
--WjW