ipsec tunnel and vnet jails: routing, howto?

[Michael Grimm]

Julian Elischer <julian at freebsd.org> wrote:
> 
> On 27/12/2015 4:24 AM, Michael Grimm wrote:

>> I am currently stuck, somehow, and I do need your input. Thus, let me explain, what I do want to achieve:
>> 
>> I do have two servers connected via an ipsec/tunnel ...
>> 	[A] dead:beef:1234:abcd::1 <—> dead:feed:abcd:1234::1 [B]
>> … which is sending all traffic destined for dead:beef:1234:abcd::/64 and dead:feed:abcd:1234::/64 through the tunnel, and vice versa.
>> 
>> That did run perfectly well during the last years until I decided to give VNET jails a try. Previously, some of my old fashioned jails got an IPv6 address attached like dead:beef:1234:abcd:1:2::3, and I could reach that address from the remote server without any routing/re-directing or alike, necessary. Now, after having moved those jails to VNET jails (having those addresses bound to their epairXXb interfaces), I cannot reach those addresses within those jails any longer.
>> 
>> >From my point of view and understanding this must have to do with lack of proper routing, but I am not sure, if that is correct, thus my questions to the experts:
>> 
>> 1) Is my assumption correct, that my tunnel is "ending" after having passed my firewalls at each server, *bevor* decrypting its ESP traffic into its final destination (yes, I do have pf rules to allow for esp traffic to pass my outer internet facing interface)?
>> 
>> 2) If that is true, racoon has to decide where to deliver those packets, finally?
>> 
>> 3) If that is true, I do have an issue with routing that *cannot* be solved by pf firewall rules, right?
>> 
>> 4) If that is true, what do I have to look for? What am I missing? How can I route incoming and finally decrypted traffic to its final destination within a VNET jail?
>> 
>> 5) Do I need to look for a completely different approach? Every hint is highly welcome.
> 
> basically you have to treat the jails as if they are totally separate machines that are reached through the vpn endpoints instead of being the endpoints themselves.
> This will require a different setup.  for example your tunnel will need to be exactly that  a tunnel and not just an encapsulation. And you will need full routing information for the other end at each end.

Thanks for your input. In the meantime I got it running, somehow. The "somehow" refers to: I am not sure if that's the way its supposed to be.

What I did (I do only show the part of host [A], the other host is configured accordingly):

1. ipsec/tunnel between [A] dead:beef:1234:abcd::1 <—> dead:feed:abcd:1234::1 [B]

   /path-to-racoon/setkey.conf:
	spdadd dead:beef:1234:abcd::/56 dead:feed:abcd:1234:1:2::3 any -P out ipsec esp/tunnel/dead:beef:1234:abcd::1-dead:feed:abcd:1234::1/require; 
	spdadd dead:feed:abcd:1234::/56 dead:beef:1234:abcd:1:2::3 any -P in  ipsec esp/tunnel/dead:feed:abcd:1234::1-dead:beef:1234:abcd::1/require;

2. routing at [A]:

   /etc/rc.conf:
	ipv6_static_routes="jail1" 
	# that's for the route from host system [A] into jail1 with IPv6 address of fd00:ffff:ffff:ffff:aaaa::1
—>	ipv6_route_mail="-host dead:beef:1234:abcd:1:2::3 -host fd00:ffff:ffff:ffff:aaaa::1"
 
  /etc/jail.conf:
	#
	# host dependent global settings
	#
	$ip6prefix		 = "dead:beef:1234:abcd";
	$ip6prefix_remote_host	 = "dead:feed:abcd:1234";

	#
	# global jail settings
	#
	host.hostname		 = "${name}";
	path			 = "/usr/home/jails/${name}";
	mount.fstab		 = "/etc/fstab.${name}";
	exec.consolelog 	 = "/var/log/jail_${name}_console.log";
	vnet			 = "new";
	vnet.interface		 = "epair${jailID}b";
	exec.clean;
	mount.devfs;
	persist;
	#
	# network settings to apply/destroy during start/stop of every jail
	#
	exec.prestart		 = "sleep 2";
	exec.prestart		+= "ifconfig epair${jailID} create up";
	exec.prestart		+= "ifconfig bridge0 addm epair${jailID}a";
	exec.start		 = "/sbin/ifconfig lo0 127.0.0.1 up";
	exec.start		+= "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
	exec.start		+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
	exec.start		+= "/sbin/route add default -gateway 10.x.x.254";
	exec.start		+= "/sbin/route add -inet6 default -gateway fd00:ffff:ffff:ffff:aaaa::254";
	exec.stop		 = "/bin/sh /etc/rc.shutdown";
	exec.poststop 		 = "ifconfig epair${jailID}a destroy";
	#
	# individual jail settings
	#
	mail {
		$jailID		 = 1;
		$ip4_addr	 = 10.x.x.1;
		$ip6_addr	 = fd00:ffff:ffff:ffff:aaaa::1/64;
		exec.start	+= "/sbin/ifconfig epair${jailID}b inet6 ${ip6prefix}:1:2::3/56 alias";
—>	# that's for the route to remote host dead:feed:abcd:1234:1:2::3 at tunnel end point [B] out of jail1
		exec.start	+= "/sbin/route add -6 ${ip6prefix_remote_host}:1:2::3 fd00:ffff:ffff:ffff:aaaa::254";
		exec.start	+= "/bin/sh /etc/rc";
	}

That is working well, after racoon has established the tunnel. 

*But* unlikely what I have observed before, the very first contact to the remote server's [B] jail out of a jail at [A] doesn't trigger racoon to establish the tunnel. Before, that happened instantaneously, but now I do need to to some "tricks" with ping6s and/or restarting racoon at the host system. I haven't found out yet what the cause is … I am sure that I need to learn much more regarding routing. Every feedback is highly welcome.

Thanks and regards,
Michael