Have I got this VIMAGE setup correct?

[Matthew D. Fuller]

On Tue, Dec 22, 2015 at 12:05:07PM -0500 I heard the voice of
Garrett Wollman, and lo! it spake thus:
>
> The consensus when I asked seemed to be that VIMAGE+jail was the
> right combination to give every container its own private loopback
> interface, so I tried to build that.  I noticed a few things:

I've got a server running a dozen or so VIMAGE jails, so I can at
least chime in a little...


> 1) The kernel prints out a warning message at boot time that VIMAGE
> is "highly experimental".  Should I be concerned about running this
> in production?

It hasn't blown up anything for me yet.


> 2) Stopping jails with virtual network stacks generates warnings from
> UMA about memory being leaked.

I'm given to understand that's Known, and presumably Not Quite Trivial
To Fix.  Since I'm not starting/stopping jails repeatedly as a normal
runtime thing, I'm ignoring it.  If you were spinning jails up and
down dynamically dozens of times a day, I'd want to look more closely
at just what is leaking and why...


> 3) It wasn't clear (or documented anywhere that I could see) how to
> get the host network set up properly.  Obviously I'm not going to
> have a vlan for every single jail, so it seemed like what most
> people were doing was "bridge" along with a bunch of "epair"
> interfaces.  I ended up with the following:

Is what I'm doing, though I'm creating the epair's and adding them to
the bridges in the setup script rather than rc.conf (exec.prestart in
jail.conf), because that makes it a more manageable IME, and since I'm
already doing a bunch of setup in the script anyway...


> In each of the jails I have to manually configure a MAC address
> using /etc/start_if.epairNb to ensure that it's globally unique, but
> then everything seems to work.

I hardcode (well, dynamically generated hardcoded) MAC addresses on
the epair's in the setup script, since
<https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=184149> bit me hard
when I was first setting it up.


-- 
Matthew Fuller     (MF4839)   |  fullermd at over-yonder.net
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
           On the Internet, nobody can hear you scream.