Running jails on multiple subnets with multiple interfaces

[Julian Elischer]

Josh Paetzel wrote:
> Jeffrey Williams wrote:
>> I have a server with two interfaces, I want to run the host and a couple of 
>> jails using one interface on one subnet (internal interface, private IP, behind 
>> NAT/firewall) and some other jails using the other interface on another subnet 
>> (external interface, public IP, DMZ).
>>
>> Now my understanding of the challenge in doing this, is that the network stack 
>> is not "virtualized" in the jails, so all the jails use the same routing table, 
>> and for obvious reasons only one default router. (also just for sake of clarity 
>> I don't want to enable routing between interfaces on the jail host)
>>
>> Now if I understand all this correctly, then what will happen is, if I set the 
>> default router to the internal networks exit router (the NAT/firewall), then 
>> the jails listening on the external interface will only be able to talk to 
>> their local subnet, and because the internal subnet won't exist for them they 
>> won't be able to connect to the network at large.
>>
>> If I set the default router to the external networks exit router (the DMZ 
>> perimeter firewall) then the host and jails listening on the internal network 
>> won't be able to be able to talk to the internet beyond the local nets, the 
>> jails because the external network doesn't exist for them, and the host because 
>> even though it can talk to both nets, the services are configured to only 
>> listen to the internal net, and the it will be trying to send all outgoing 
>> traffic to the public net, thus not creating and NAT table entries on the 
>> NAT/Firewall for the return connections.
>>
>> Is there anyway to achieve what I have trying to do?
>>
>> Thanks
>> Jeffrey williams
> 
> PF makes a very effective workaround to this with it's route-to
> option...effectively letting you bypass the routing table altogether
> and set up per IP behavior.
> 
> For instance, I use it in the following scenario, where a box has two
> interfaces with public IPs and I don't want answers to connections on
> the 'secondary' interface to go out the default route.

ipfw can also do this using the fwd rule.
in 7.x (and 6-stable) you can also do:

ipfw table 1 add 1.2.3.4/28 2.2.2.2   <-- a specific route
ipfw table 1 add 0.0.0.0/0 3.3.3.3    <-- a default route

ipfw add 300 fwd tablearg ip from ${ADDRESS2} to table(1) out


> 
> connection 1's router 192.168.1.1
> em0 ip 192.168.1.2/24
> 
> connection 2's router 10.0.0.1
> em1 ip 10.0.0.2/24
> 
> if connection 1 is the 'primary' link then set the default route to
> 192.168.1.1 and put the following rule in pf.conf
> 
> pass out route-to (em1 10.0.0.1) from 10.0.0.2 to ! 10.0.0.0/24
> 
> If you were to give more concrete examples of your config I could
> probably help you out with a workable pf solution.
>