deploy multiple vnets with VIMAGE/VNET + Production Ready?

[Michael Grimm]

Sebastián Maruca via freebsd-jail <freebsd-jail at freebsd.org> wrote:

> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one…

Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin. 


Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs:

1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.:

	/etc/fstab.www
		/path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0 

2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x

3) Networking regarding VNET jails defined in /etc/rc.conf:

	# set up one bridge interface
	cloned_interfaces="bridge0"

	# needed for default routes within jails
	ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
	ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"

4) Thus, jails are controlled by jail(8) (shown for 3 example jails):

	/etc/rc.conf
		———————————————BEGIN------------------------
		jail_enable="YES"
		jail_reverse_stop="YES"
		jail_list="dns www mail"
		———————————————-END————————————

	/etc/jail.conf:
		#
		# host dependent global settings
		#
		$ip6prefixLOCAL		 = "fd00:dead:dead:beef";
		
		#
		# global jail settings
		#
		host.hostname		 = "${name}";
		path			 = "/path-to-your/jails/${name}";
		mount.fstab		 = "/etc/fstab.${name}";
		exec.consolelog 	 = "/var/log/jail_${name}_console.log";
		vnet			 = "new";
		vnet.interface		 = "epair${jailID}b";
		exec.clean;
		mount.devfs;
		persist;
		
		#
		# network settings to apply/destroy during start/stop of every jail
		#
		exec.prestart		 = "sleep 2";
		exec.prestart		+= "ifconfig epair${jailID} create up";
		exec.prestart		+= "ifconfig bridge0 addm epair${jailID}a";
		exec.start		 = "/sbin/ifconfig lo0 127.0.0.1 up";
		exec.start		+= "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
		exec.start		+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
		exec.start		+= "/sbin/route add default -gateway 10.1.1.254";
		exec.start		+= "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254";
		#exec.stop		 = "/sbin/route del default";
		#exec.stop		+= "/sbin/route del -inet6 default";
		exec.stop		+= "/bin/sh /etc/rc.shutdown";
		exec.poststop 		 = "ifconfig epair${jailID}a destroy";
		
		#
		# individual jail settings
		#
		mail {
			$jailID		 = 1;
			$ip4_addr	 = 10.1.1.1;
			$ip6_addr	 = ${ip6prefixLOCAL}::1/64;
			exec.start	+= "/bin/sh /etc/rc";
		}
		
		www {
			$jailID		 = 2;
			$ip4_addr	 = 10.1.1.2;
			$ip6_addr	 = ${ip6prefixLOCAL}::2/64;
			exec.start	+= "/bin/sh /etc/rc";
		}
		
		dns {
			$jailID		 = 3;
			$ip4_addr	 = 10.1.1.3;
			$ip4_addr_2	 = 10.1.1.4;
			$ip6_addr	 = ${ip6prefixLOCAL}::3/64;
			$ip6_addr_2	 = ${ip6prefixLOCAL}::4/64;
			exec.start	+= "/sbin/ifconfig epair${jailID}b inet  ${ip4_addr_2} alias";
			exec.start	+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias";
			exec.start	+= "/bin/sh /etc/rc";
		}
		
	Now you can use "service jail" to start/stop your jails, e.g.:

		service jail stop 
		service jail restart dns
		service jail start dns mail

5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"

	#!/bin/csh
	
	#
	# restart jail services without removing jail and its network
	#
	
	#
	# global definitions
	#
	set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
	set RCDIR = "/usr/local/etc/rc.d"
	set TAB = "        "
	
	#
	# evaluate list of rc files in /usr/local/etc/rc.d
	#
	set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
	
	#
	# evaluate reverse order of RCFILES
	#
	set RCFILES_REVERSE = ""
	foreach rcname ( ${RCFILES} )
		set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}"
	end
	
	#
	# stop rc services
	#
	echo "stopping:"
	foreach rcname ( ${RCFILES_REVERSE} )
		${LOGGER} stopping ${rcname}
		${rcname} stop >& /dev/null
		echo "${TAB}" ${rcname}
	end
	
	#
	# start rc services
	#
	echo "starting:"
	foreach rcname ( ${RCFILES} )
		${LOGGER} starting ${rcname}
		${rcname} start >& /dev/null
		echo "${TAB}" ${rcname}
	end
	
	exit 0

This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-)

I hope that helps for a start. Again, I am sure you may need some tweaking at your site.

Regards,
Michael