IPSec tunnel, VNET jail and routing issue

[Michael Grimm]

Hi --

I am referring to the following (simplified) setup:

[hostA /ix0 / 2001:dead::1 / 1.2.3.4] <===== IPsec tunnel =====> [hostB / ix0 / 2001:beef::10 / 10.20.30.40]
   |                                                                |     
   |                                                                |
[jail1 / bridge0 / fd00:a::1 / 10.1.1.1]                         [jail1 / bridge0 / fd00:b::2 / 10.2.2.2]

All my jails are VNET jails, that use the bridge0 (epair) device. Thus, all IPv4 and IPv6 addresses of my local networks an A and B are bound to the bridge0 interface! But, the IPsec tunnel (via racoon) is anchored at public IPv4 addresses on ix0 at both hosts.

Task:
	route all local traffic from hostA to hostB via the tunnel.

Working:
	IPv6 traffic is running fine, meaning, that I can reach every jail from every host. That has been working for years.

Issue:
	I recently wanted to extend my setup to local IPv4 addresses of my jails, and failed miserably.

Configuration (shown for hostA, only):

	setkey.conf
		#      hostA          hostB                                       hostA   hostB 
		spdadd fd00:a::/64    fd00:b::/64     any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		spdadd fd00:a::/64    2001:beef::/56  any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		spdadd 2001:dead::/56 fd00:b::/64     any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		#      hostB          hostA                                       hostB       hostA 
		spdadd fd00:b::/64    fd00:a::/64     any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;
		spdadd fd00:b::/64    2001:dead::/56  any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;
		spdadd 2001:beef::/56 fd00:a::/64     any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;
		#      hostA          hostB                                       hostA   hostB 
		spdadd 10.1.1.0/24    10.2.2.0/24     any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		spdadd 10.1.1.0/24    10.20.30.40     any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		spdadd 1.2.3.4        10.2.2.0/24     any -P out ipsec esp/tunnel/1.2.3.4-10.20.30.40/require;
		#      hostB          hostA                                       hostB       hostA 
		spdadd 10.2.2.0/24    10.1.1.0/24     any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;
		spdadd 10.2.2.0/24    1.2.3.4         any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;
		spdadd 10.20.30.40    10.1.1.0/24     any -P in  ipsec esp/tunnel/10.20.30.40-1.2.3.4/require;

	There is no specific routing regarding the tunnel defined. All should be done by this spdadd's.

Achieved sofar:

	#) I can reach each jail at the other site from the host.

	#) Allowing arpproxy_all="YES" will satisfy ARP (MACs from opposite VNET jails will become assigned). 
           I do not know if that is needed, but now ping from jails to the opposite jails will at least start to send ICMP packages.

Unsolved issue:

	I cannot reach opposite jails from another host's jail; e.g.: ping 10.20.30.40 in jail1 at hostA will not work.

Observations so far:

	#) tcpdump shows for "ping 10.2.2.2 in jail1 at hostA" ICMP traffic at the bridge0 at hostA:
		IP 10.1.1.1 > 10.2.2.2: ICMP echo request, id 20099, seq 0, length 64 
	   and at bridge0 at hostB:
		IP 10.1.1.1 > 10.2.2.2: ICMP echo request, id 15233, seq 6, length 64 
		IP 10.2.2.2 > 10.1.1.1: ICMP echo reply, id 15233, seq 6, length 64 
		
	Hmm: hostA doesn't get an echo replay, although hostB did send one. 

	#) tcpdump shows for "ping 10.2.2.2 at hostA" *no* ICMP traffic at hostA at bridge0 or hostA at ix0 but ICMP traffic at hostB at bridge0:
		IP 1.2.3.4 > 10.2.2.2: ICMP echo request, id 60543, seq 0, length 64 
		IP 10.2.2.2 > 1.2.3.4: ICMP echo reply, id 60543, seq 0, length 64 
	
	Hmm: it's working.

	#) It looks to me as if the tunnel does not recognise "spdadd 10.1.1.0/24 10.2.2.0/24" and vice versa settings because those IPs are bound to the bridge.

	#) Whenever an IP bound to ix0 is involved (host to jail) the corresponding spdadd parts are recognised.

	#) adding static routes like "add route 10.2.2.0/24 1.2.3.4" and alike do not solve my issue. 

Questions:

	#) Is this an issue with IPsec/racoon?
	#) Is this a routing issue?
	#) Why does IPv6 address space work (identical configuration regarding jails, firewalling, routing, et al.)
	#) Any other idea?

Sorry for this lengthy post, and any feedback is highly welcome,
Michael