deploy multiple vnets with VIMAGE/VNET + Production Ready?

Sebastián Maruca juanperiz at yahoo.com.ar
Thu Jun 2 21:29:34 UTC 2016


Michael... even though you consider yourself as a admin hobbier, I can tell you have the "lend hander" top grade you're honored ;)
I'll start from this big step you're posting (and all the other which replied too) and carry on dancing 'til I got my jails running DMZ, VLAN and WAN like a pro...
Best Regards,Seba

      De: Michael Grimm <trashcan at ellael.org>
 Para: "freebsd-jail at freebsd.org" <freebsd-jail at freebsd.org> 
 Enviado: Jueves, 2 de junio, 2016 15:24:34
 Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
   
Sebastián Maruca via freebsd-jail <freebsd-jail at freebsd.org> wrote:

> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one…

Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin. 


Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs:

1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.:

    /etc/fstab.www
        /path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0 

2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x

3) Networking regarding VNET jails defined in /etc/rc.conf:

    # set up one bridge interface
    cloned_interfaces="bridge0"

    # needed for default routes within jails
    ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
    ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"

4) Thus, jails are controlled by jail(8) (shown for 3 example jails):

    /etc/rc.conf
        ———————————————BEGIN------------------------
        jail_enable="YES"
        jail_reverse_stop="YES"
        jail_list="dns www mail"
        ———————————————-END————————————

    /etc/jail.conf:
        #
        # host dependent global settings
        #
        $ip6prefixLOCAL        = "fd00:dead:dead:beef";
        
        #
        # global jail settings
        #
        host.hostname        = "${name}";
        path            = "/path-to-your/jails/${name}";
        mount.fstab        = "/etc/fstab.${name}";
        exec.consolelog     = "/var/log/jail_${name}_console.log";
        vnet            = "new";
        vnet.interface        = "epair${jailID}b";
        exec.clean;
        mount.devfs;
        persist;
        
        #
        # network settings to apply/destroy during start/stop of every jail
        #
        exec.prestart        = "sleep 2";
        exec.prestart        += "ifconfig epair${jailID} create up";
        exec.prestart        += "ifconfig bridge0 addm epair${jailID}a";
        exec.start        = "/sbin/ifconfig lo0 127.0.0.1 up";
        exec.start        += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
        exec.start        += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
        exec.start        += "/sbin/route add default -gateway 10.1.1.254";
        exec.start        += "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254";
        #exec.stop        = "/sbin/route del default";
        #exec.stop        += "/sbin/route del -inet6 default";
        exec.stop        += "/bin/sh /etc/rc.shutdown";
        exec.poststop         = "ifconfig epair${jailID}a destroy";
        
        #
        # individual jail settings
        #
        mail {
            $jailID        = 1;
            $ip4_addr    = 10.1.1.1;
            $ip6_addr    = ${ip6prefixLOCAL}::1/64;
            exec.start    += "/bin/sh /etc/rc";
        }
        
        www {
            $jailID        = 2;
            $ip4_addr    = 10.1.1.2;
            $ip6_addr    = ${ip6prefixLOCAL}::2/64;
            exec.start    += "/bin/sh /etc/rc";
        }
        
        dns {
            $jailID        = 3;
            $ip4_addr    = 10.1.1.3;
            $ip4_addr_2    = 10.1.1.4;
            $ip6_addr    = ${ip6prefixLOCAL}::3/64;
            $ip6_addr_2    = ${ip6prefixLOCAL}::4/64;
            exec.start    += "/sbin/ifconfig epair${jailID}b inet  ${ip4_addr_2} alias";
            exec.start    += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias";
            exec.start    += "/bin/sh /etc/rc";
        }
        
    Now you can use "service jail" to start/stop your jails, e.g.:

        service jail stop 
        service jail restart dns
        service jail start dns mail

5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"

    #!/bin/csh
    
    #
    # restart jail services without removing jail and its network
    #
    
    #
    # global definitions
    #
    set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
    set RCDIR = "/usr/local/etc/rc.d"
    set TAB = "        "
    
    #
    # evaluate list of rc files in /usr/local/etc/rc.d
    #
    set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
    
    #
    # evaluate reverse order of RCFILES
    #
    set RCFILES_REVERSE = ""
    foreach rcname ( ${RCFILES} )
        set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}"
    end
    
    #
    # stop rc services
    #
    echo "stopping:"
    foreach rcname ( ${RCFILES_REVERSE} )
        ${LOGGER} stopping ${rcname}
        ${rcname} stop >& /dev/null
        echo "${TAB}" ${rcname}
    end
    
    #
    # start rc services
    #
    echo "starting:"
    foreach rcname ( ${RCFILES} )
        ${LOGGER} starting ${rcname}
        ${rcname} start >& /dev/null
        echo "${TAB}" ${rcname}
    end
    
    exit 0

This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-)

I hope that helps for a start. Again, I am sure you may need some tweaking at your site.

Regards,
Michael





_______________________________________________
freebsd-jail at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscribe at freebsd.org"

  


More information about the freebsd-jail mailing list