deploy multiple vnets with VIMAGE/VNET + Production Ready?
Sebastián Maruca
juanperiz at yahoo.com.ar
Thu Jun 2 21:29:34 UTC 2016
Michael... even though you consider yourself as a admin hobbier, I can tell you have the "lend hander" top grade you're honored ;)
I'll start from this big step you're posting (and all the other which replied too) and carry on dancing 'til I got my jails running DMZ, VLAN and WAN like a pro...
Best Regards,Seba
De: Michael Grimm <trashcan at ellael.org>
Para: "freebsd-jail at freebsd.org" <freebsd-jail at freebsd.org>
Enviado: Jueves, 2 de junio, 2016 15:24:34
Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
Sebastián Maruca via freebsd-jail <freebsd-jail at freebsd.org> wrote:
> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one…
Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin.
Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs:
1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.:
/etc/fstab.www
/path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0
2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x
3) Networking regarding VNET jails defined in /etc/rc.conf:
# set up one bridge interface
cloned_interfaces="bridge0"
# needed for default routes within jails
ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"
4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
/etc/rc.conf
———————————————BEGIN------------------------
jail_enable="YES"
jail_reverse_stop="YES"
jail_list="dns www mail"
———————————————-END————————————
/etc/jail.conf:
#
# host dependent global settings
#
$ip6prefixLOCAL = "fd00:dead:dead:beef";
#
# global jail settings
#
host.hostname = "${name}";
path = "/path-to-your/jails/${name}";
mount.fstab = "/etc/fstab.${name}";
exec.consolelog = "/var/log/jail_${name}_console.log";
vnet = "new";
vnet.interface = "epair${jailID}b";
exec.clean;
mount.devfs;
persist;
#
# network settings to apply/destroy during start/stop of every jail
#
exec.prestart = "sleep 2";
exec.prestart += "ifconfig epair${jailID} create up";
exec.prestart += "ifconfig bridge0 addm epair${jailID}a";
exec.start = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
exec.start += "/sbin/route add default -gateway 10.1.1.254";
exec.start += "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254";
#exec.stop = "/sbin/route del default";
#exec.stop += "/sbin/route del -inet6 default";
exec.stop += "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair${jailID}a destroy";
#
# individual jail settings
#
mail {
$jailID = 1;
$ip4_addr = 10.1.1.1;
$ip6_addr = ${ip6prefixLOCAL}::1/64;
exec.start += "/bin/sh /etc/rc";
}
www {
$jailID = 2;
$ip4_addr = 10.1.1.2;
$ip6_addr = ${ip6prefixLOCAL}::2/64;
exec.start += "/bin/sh /etc/rc";
}
dns {
$jailID = 3;
$ip4_addr = 10.1.1.3;
$ip4_addr_2 = 10.1.1.4;
$ip6_addr = ${ip6prefixLOCAL}::3/64;
$ip6_addr_2 = ${ip6prefixLOCAL}::4/64;
exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2} alias";
exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias";
exec.start += "/bin/sh /etc/rc";
}
Now you can use "service jail" to start/stop your jails, e.g.:
service jail stop
service jail restart dns
service jail start dns mail
5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"
#!/bin/csh
#
# restart jail services without removing jail and its network
#
#
# global definitions
#
set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
set RCDIR = "/usr/local/etc/rc.d"
set TAB = " "
#
# evaluate list of rc files in /usr/local/etc/rc.d
#
set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
#
# evaluate reverse order of RCFILES
#
set RCFILES_REVERSE = ""
foreach rcname ( ${RCFILES} )
set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}"
end
#
# stop rc services
#
echo "stopping:"
foreach rcname ( ${RCFILES_REVERSE} )
${LOGGER} stopping ${rcname}
${rcname} stop >& /dev/null
echo "${TAB}" ${rcname}
end
#
# start rc services
#
echo "starting:"
foreach rcname ( ${RCFILES} )
${LOGGER} starting ${rcname}
${rcname} start >& /dev/null
echo "${TAB}" ${rcname}
end
exit 0
This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-)
I hope that helps for a start. Again, I am sure you may need some tweaking at your site.
Regards,
Michael
_______________________________________________
freebsd-jail at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscribe at freebsd.org"
More information about the freebsd-jail
mailing list