VIMAGE: vnet, epair and lots of jails on bridgeX - routing

[O. Hartmann]

On Mon, 12 Feb 2018 09:37:47 +0100
"O. Hartmann" <ohartmann at walstatt.org> wrote:

> On Sat, 10 Feb 2018 11:52:18 +0100
> Olivier Cochard-Labbé <olivier at freebsd.org> wrote:
> 
> > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann <ohartmann at walstatt.org> wrote:
> >   
> > >
> > > The moment any of the bridges gets an additional member epair interface
> > > (so the bridge
> > > has at least three members including the on reaching into the virtual
> > > router jail) the
> > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of
> > > that vbridge
> > > are unreachable.
> > >
> > >    
> > ​First idea:
> > Did you try with a more simple setup, like with just 3 jails members of one
> > bridge ?  
> > => I've tried it on a -head, and all 4 members  (3 jails and the host)    
> > reach to communicate.
> > 
> > Second idea:
> > Can you check that all epairs have different MAC address each ?​
> > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671  
> 
> Wow, that PR is from 2013(!) and it is still not solved?
> 
> > 
> > Regards,
> > 
> > Olivier
> > _______________________________________________
> > freebsd-current at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"  
> 
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"

After rebooting recent CURRENT, the view with "ifconfig -a ether" looked good
so far, each epair/bridge has its unique MAC.

But then, login on the jails and checking the epair's counterpart owned by the
VIMAGE jail, I found almost EVERY jail has the same MAC, even those jails
members of the same bridge:

[...]
jail 11  on bridge2
epair20129a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500 options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

jail 10 on bridge2
epair20128a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500 options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

jail 9 on bridge 1
epair10250a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500 options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

jail 8 on bridge1
epair10238a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500 options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

jail 7 on bridge0
epair238a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

epair251a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:68:d0:00:07:0a

The way I create epairs and put them into a jail's context/domain is straight
forward. In jail.conf, I have more generic setup with variables like:

# DNS Master
ns1 {
        $if =                   "2";
        $ip4_addr =             "10.10.0.${if}";
        $ip4_cidr =             "24";
        $ip4_my_default_route = "10.10.0.1";
        $vnet_if =              "epair${if}";
        $home_bridge =          "${if_bridge_core}";

        depend=                 "vrouter";

        allow.raw_sockets=      "1";
}


and in the common portion of jail.conf definitions, I use this:

[...]

vnet =                  "new";
vnet.interface =        "${vnet_if}a";
persist;

exec.clean;

exec.start=             "";
exec.start+=            "/sbin/ifconfig ${vnet_if}a inet
${ip4_addr}/${ip4_cidr} up"; exec.start+=            "/bin/sh /etc/rc";
exec.start+=            "/sbin/route add default ${ip4_my_default_route}";
exec.start+=            "/sbin/sysctl net.link.bridge.pfil_member=0";
exec.start+=            "/sbin/sysctl net.link.bridge.pfil_bridge=0";
exec.start+=            "/sbin/sysctl net.link.bridge.pfil_onlyip=0";

exec.stop=              "/bin/sh /etc/rc.shutdown";

exec.prestart=          "";
exec.prestart+=         "ifconfig ${vnet_if} create";
exec.prestart+=         "ifconfig ${vnet_if}b up";
exec.prestart+=         "ifconfig ${home_bridge} addm ${vnet_if}b up";

exec.poststop=          "ifconfig ${home_bridge} deletem ${vnet_if}b";
exec.poststop+=         "ifconfig ${vnet_if}b destroy";

exec.consolelog=        "/var/log/jail_${name}_console.log";


The big question here is: when a jail with VIMAGE kernel "swallows" a
epair-pseudo device, it leaves the ciontext or visibility of the host. How can
the FreeBSD VIMAGE kernel then know about what former epair's MAC was? Is this
mechanism maybe the culprit? It is just a thought, so I do not want to be
beheaded - I'm not much into system development.


Kind regards,

O. Hartmann