Multiple Subnets/interfaces

[Panagiotis Christias]

On Fri, Nov 26, 2010 at 12:55 AM, Andrew Hotlab
<andrew.hotlab at hotmail.com> wrote:
>
>> From: danieljfaulknor at gmail.com
>> Date: Wed, 24 Nov 2010 16:00:52 +1300
>> To: freebsd-jail at freebsd.org
>> Subject: Multiple Subnets/interfaces
>>
>> Hi,
>>
>> I have a FreeBSD server with two interfaces, both on different physical networks. I want to have some jails on each network.
>>
>> # This file now contains just the overrides from /etc/defaults/rc.conf.
>> hostname="fs1.akl1.generatornz.com"
>> ifconfig_bce0="inet 120.xxx.52.2/28"
>> ifconfig_bce0_alias0="inet 120.xxx.52.3/32"
>> ifconfig_bce0_alias1="inet 120.xxx.52.4/32"
>> ifconfig_bce0_alias2="inet 120.xxx.52.6/32"
>> ifconfig_bce1="inet 120.xxx.55.146/25"
>> ifconfig_bce1_alias0="inet 120.xxx.55.147/25"
>> defaultrouter="120.xxx.52.1"
>>
>> I have added this to my pf:
>>
>> pass out route-to ($ext_if 120.xxx.55.129) from 120.xxx.55.147 to ! 120.xxx.55.128/23
>>
>
> This rule seems correct if you want to allow only the IP 120.xxx.55.147 to be reachable by hosts outside the subnet,
> even if the CIDR notation you wrote at the end of the line seems wrong to me. Shouldn't it be 120.xxx.55.128/25 ?
> I think that it is a best practice to add the secondary IP address to the bge1 interface with a /32 netmask.
>
>> I can ping 120.xxx.55.147, but i can't connect to any services that netstat is showing that is running:
>>
>> Active Internet connections (including servers)
>> Proto Recv-Q Send-Q  Local Address          Foreign Address       (state)
>> tcp4       0      0 ip-120-136-55-14.ssh   *.*                    LISTEN
>> tcp4       0      0 ip-120-136-55-14.http  *.*                    LISTEN
>> tcp4       0      0 ip-120-136-55-14.9000  *.*                    LISTEN
>>
>
> In the past I used such PF rules to enable several jails on the same host in different subnets to reach interconnected
> networks, but I have been using the multiple FIB feature since FreeBSD 7.2.  Now I'm running the RELENG_8_1
> and I'm still very satisfied by this method, which I feel more "solid" than the PF "hack".
>
> I don't remember exactly why, but in my last functioning ruleset I've just noticed the "no state" command at the end
> of each rule.  Hope this might help you.

Same setup here, FreeBSD 7.3 server with multiple physical interfaces
(plus vlan interfaces) connected on different networks. The hosting
server has its primary IP address and default gateway on one of
connected networks. Hosted jails on other than the primary network use
the appropriate FIB in order to reach the default gateway of their
connected network. Also, for every configured IP address on the server
(jail or not) I use two ipfw rules like:

101 fwd X.Y.Z.1 icmp from X.Y.Z.10 to not X.Y.Z.0/24
102 fwd X.Y.Z.1 ip from X.Y.Z.10 to not X.Y.Z.0/24 via NIC1

where X.Y.Z.0/24 is the connected network, NIC1 is the physical
interface or vlan interface, X.Y.Z.1 is the network's default gateway
and X.Y.Z.10 is the configured IP address. These rules will ensure
that replies from X.Y.Z.10 to remote hosts on networks that our server
is directly connected on will go through the default gateway and not
directly through server's connected interface causing asymmetric
routing.

I am not sure if security.jail.ip4_saddrsel would do the trick (see
http://www.freebsd.org/releases/7.3R/relnotes-detailed.html) instead
of the ipfw rules. ICMP replies are handled directly by the server's
kernel and not by the jail subsytem.

Regards,
Panagiotis