Multihomed system with jails routing issues

[Chris Smith]

Hi All,

I have a system with 1 network interface with 2 extra VLANs off it and 
I'm having some trouble getting the routing working correctly with it 
and jails.

bge0     - management - 10.71.100.0/24
bge0.101 - LAN        - 10.71.101.0/24
bge0.103 - DMZ        - 10.71.101.0/24

Here's what I want to achieve...

Host:
I want the host system to only listen on one interface, bge0. I want NO 
ip addresses of the host on the vlan interfaces. The only service it 
will be exposing is its sshd. The management address for this system is 
10.71.100.50.

Jails:
The system will also host a variety of jails, each with an IP either on 
the LAN or DMZ. I am using ezjail to manage the jails.

Router:
There is a router at the .254 address of every subnet that can route 
between each network.

I set up jail1 on bge0.101 with the IP 10.71.101.51. Since the host does 
not have an address configured on bge0.101, I configured the jail 
address as /24 instead of the default /32.

My issues:

* If I do not configure the jail as a /24 (e.g. /32), the LAN cannot 
communicate with the jail.

* When the jail is up and 10.71.101.51/24 is active, SSHing from the LAN 
to the mgmt interface via the router fails, as the host tries to send 
return traffic via the bge0.101 interface, even though traffic arrived 
via the bge0 interface.

So I did a whole lot of research for people having these apparently 
problems, and decided to try the multiple routing table/fib approach. So 
I recompiled my kernel, configured fib 1 with the LAN interface route 
(setfib route add 10.71.101.0/24 -iface bge0.101), set the jail fib and 
set the tunable net.addr_all_fibs = 0. I still can't get this working 
correctly. ezjail still seems to add the interface route to fib 0 by 
default (but it won't if i run ezjail with the setfib 1 command).

Using FIB 1 and trying to ping hosts on the LAN gives an error like: 
sendto failed: invalid argument.

Does anybody have any best practices for doing this, or anything else I 
can try? I'm happy to share/pastebin any configuration and I've tried 
most things I've found on the internet. I'm using FreeBSD 10.0 with a 
custom kernel for multiple routing tables.

Thanks in advance!
Chris.