deploy multiple vnets with VIMAGE/VNET + Production Ready?

[Ernie Luzar]

Michael Grimm wrote:
> Sebastián Maruca via freebsd-jail <freebsd-jail at freebsd.org> wrote:
> 
>> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge...
>> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one…
> 
> Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin. 
> 
> 
> Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs:
> 
> 1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.:
> 
> 	/etc/fstab.www
> 		/path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0 
> 
> 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x
> 
> 3) Networking regarding VNET jails defined in /etc/rc.conf:
> 
> 	# set up one bridge interface
> 	cloned_interfaces="bridge0"
> 
> 	# needed for default routes within jails
> 	ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
> 	ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"
> 
> 4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
> 
> 	/etc/rc.conf
> 		———————————————BEGIN------------------------
> 		jail_enable="YES"
> 		jail_reverse_stop="YES"
> 		jail_list="dns www mail"
> 		———————————————-END————————————
> 
> 	/etc/jail.conf:
> 		#
> 		# host dependent global settings
> 		#
> 		$ip6prefixLOCAL		 = "fd00:dead:dead:beef";
> 		
> 		#
> 		# global jail settings
> 		#
> 		host.hostname		 = "${name}";
> 		path			 = "/path-to-your/jails/${name}";
> 		mount.fstab		 = "/etc/fstab.${name}";
> 		exec.consolelog 	 = "/var/log/jail_${name}_console.log";
> 		vnet			 = "new";
> 		vnet.interface		 = "epair${jailID}b";
> 		exec.clean;
> 		mount.devfs;
> 		persist;
> 		
> 		#
> 		# network settings to apply/destroy during start/stop of every jail
> 		#
> 		exec.prestart		 = "sleep 2";
> 		exec.prestart		+= "ifconfig epair${jailID} create up";
> 		exec.prestart		+= "ifconfig bridge0 addm epair${jailID}a";
> 		exec.start		 = "/sbin/ifconfig lo0 127.0.0.1 up";
> 		exec.start		+= "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
> 		exec.start		+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
> 		exec.start		+= "/sbin/route add default -gateway 10.1.1.254";
> 		exec.start		+= "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254";
> 		#exec.stop		 = "/sbin/route del default";
> 		#exec.stop		+= "/sbin/route del -inet6 default";
> 		exec.stop		+= "/bin/sh /etc/rc.shutdown";
> 		exec.poststop 		 = "ifconfig epair${jailID}a destroy";
> 		
> 		#
> 		# individual jail settings
> 		#
> 		mail {
> 			$jailID		 = 1;
> 			$ip4_addr	 = 10.1.1.1;
> 			$ip6_addr	 = ${ip6prefixLOCAL}::1/64;
> 			exec.start	+= "/bin/sh /etc/rc";
> 		}
> 		
> 		www {
> 			$jailID		 = 2;
> 			$ip4_addr	 = 10.1.1.2;
> 			$ip6_addr	 = ${ip6prefixLOCAL}::2/64;
> 			exec.start	+= "/bin/sh /etc/rc";
> 		}
> 		
> 		dns {
> 			$jailID		 = 3;
> 			$ip4_addr	 = 10.1.1.3;
> 			$ip4_addr_2	 = 10.1.1.4;
> 			$ip6_addr	 = ${ip6prefixLOCAL}::3/64;
> 			$ip6_addr_2	 = ${ip6prefixLOCAL}::4/64;
> 			exec.start	+= "/sbin/ifconfig epair${jailID}b inet  ${ip4_addr_2} alias";
> 			exec.start	+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias";
> 			exec.start	+= "/bin/sh /etc/rc";
> 		}
> 		
> 	Now you can use "service jail" to start/stop your jails, e.g.:
> 
> 		service jail stop 
> 		service jail restart dns
> 		service jail start dns mail
> 
> 5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www"
> 
> 	#!/bin/csh
> 	
> 	#
> 	# restart jail services without removing jail and its network
> 	#
> 	
> 	#
> 	# global definitions
> 	#
> 	set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC"
> 	set RCDIR = "/usr/local/etc/rc.d"
> 	set TAB = "        "
> 	
> 	#
> 	# evaluate list of rc files in /usr/local/etc/rc.d
> 	#
> 	set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:`
> 	
> 	#
> 	# evaluate reverse order of RCFILES
> 	#
> 	set RCFILES_REVERSE = ""
> 	foreach rcname ( ${RCFILES} )
> 		set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}"
> 	end
> 	
> 	#
> 	# stop rc services
> 	#
> 	echo "stopping:"
> 	foreach rcname ( ${RCFILES_REVERSE} )
> 		${LOGGER} stopping ${rcname}
> 		${rcname} stop >& /dev/null
> 		echo "${TAB}" ${rcname}
> 	end
> 	
> 	#
> 	# start rc services
> 	#
> 	echo "starting:"
> 	foreach rcname ( ${RCFILES} )
> 		${LOGGER} starting ${rcname}
> 		${rcname} start >& /dev/null
> 		echo "${TAB}" ${rcname}
> 	end
> 	
> 	exit 0
> 
> This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-)
> 
> I hope that helps for a start. Again, I am sure you may need some tweaking at your site.
> 
> Regards,
> Michael
> 
> 

Michael,
You left out whether you had to compile the kernel with the vimage 
option or whether vimage was already included in the kernel?