exec.fib and a jail in two subnets

[James Gritton]

On Tue, Mar 12, 2019 at 2:05 PM Grzegorz Junka <list1 at gjunka.com> wrote:
>
> On 12/03/2019 19:19, James Gritton wrote:
> > On 2019-03-10 13:40, Grzegorz Junka wrote:
> >> Hi,
> >>
> >> I am not sure if this question fits better to net or jail list so
> >> please delete one crosspost when replying.
> >>
> >> I have two routers in separate subnets (say 10.0.0.0/16 and
> >> 172.16.0.0/16). I have enabled multiple fibs on the host and I am
> >> trying to setup a jail so that packets from one router are returned to
> >> the same router. The second subnet is configured like this:
> >>
> >> setfib 1 route add -net 172.16.0.0/16 -iface lagg0
> >> setfib 1 route add default 172.16.0.1
> >>
> >> When the jail configuration is (differences in red):
> >>
> >> mta {
> >>   exec.fib=1;
> >>   ip4.addr = 172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> router 172.16.0.1 is able to send to and receive packets from the jail
> >> as expected.
> >>
> >> When the jail configuration is:
> >>
> >> mta {
> >>   ip4.addr = 10.0.0.2,172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> then router 10.0.0.1 is also able to send and receive packets from the
> >> jail as expected.
> >>
> >> However, when the configuration is:
> >>
> >> mta {
> >> exec.fib=1;
> >>   ip4.addr = 10.0.0.2,172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> then router 172.16.0.1 is no longer able to receive a response from
> >> the jail. The router's event log shows entry similar to the following
> >> two about 2 minutes apart:
> >>
> >> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP
> >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - -
> >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
> >> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP
> >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - -
> >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
> >>
> >> My question is why the 10.0.0.1 router is able to communicate with the
> >> jail in the second configuration but 172.16.0.1 is not able to
> >> communicate with the jail in the third configuration. Is it because of
> >> order of IPs in ip4.addr?
> >>
> >> When the jail is started jls shows only the first IP from either of
> >> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to
> >> 1). So my guess is that the first IP is somehow a default IP?
> >>
> >> Then my additional question is if it's possible for a jail to be in
> >> two subnets at the same time, i.e. so that when the jail responds to a
> >> packet received from router 10.0.0.1 it sends it to the default route
> >> from fib0 and when it responds to a packet received from 172.16.0.1 it
> >> sends it to the default route from fib1. What exec.fib should be in
> >> such a case?
> >>
> >> Any help would be greatly appreciated. Thanks!
> >
> > You're correct in your assumption that a jail's first IP address is
> > its default: in the absence of binding a particular address for an
> > outgoing connection, the first-listed address will be used.  So then
> > the problem with the third jail is you have a packing being sent from
> > 10.0.0.2 with only the routing table that doesn't include 10.0/16.  I
> > can't say exactly why your second example *does* work, but at least
> > from the jail side it has a default address that's reachable in its
> > routing table.  I'm thinking you're saying that the second jail works
> > not only with 10.0 but also with 172.16 (it's the 172.16 part I'm
> > unsure about).
> >
> > To answer your last question: sure, a jail can be in two subnets - but
> > it will still use its first address by default for any outbound
> > packets.  Note that the FIB associated with the jail isn't *really*
> > associated with the jail, but with the processes jail(8) starts for it
> > - the reason for the "exec" in "exec.fib". You're still free to call
> > setfib from inside the jail to access a different table.
> >
> > I haven't tried using two different routing tables in one jail at the
> > same time; the closest I've come is one jail that routed on the
> > non-default network.  Outside of the jail world, I believe multiple
> > routing tables implies multiple instances of servers, and that would
> > be the same for inside a jail.  Your router log shows port 80, so that
> > would imply two different apache (or whatever) processes running the
> > jail, each pointing to its own address, and rung under its own routing
> > table.
> >
>
> Many thanks for your response. The second example works with 10.0.0.1
> but not with 172.16.0.1, otherwise there would be no post. Following on
> your response, lets assume that a process (e.g. nginx) listens on both
> IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default
> routes or whatever so that when a packet arrives from 10.0.0.1 it is
> send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back
> to 172.16.0.1 (thus using default routes from either fib0 or fib1
> depending if the packet came from a router in one of those network)? If
> not, would it be possible to do this with some iptables/pf rules (which
> I understand in FreeBSD 12 should work in a jail with VNET)?

> My understanding (which I admit is imperfect) is that it's not
> possible with default routes alone.  At the application level, it
> would be possible if nginx was either fib-aware, or if it explicitly
> bound the source address of its replies - but neither of those are
> things typically done at the application level.


> It is possible however at the firewall level; At least I know it's
> possible for ipfw (the small corner of the firewall world that I
> use).  A quick check of ipf and ipfilter man pages didn't show "fib"
> anywhere, but don't take my word on those.  It also may require a
> VNET jail; I've never run a system with your exact setup so I'm
> unsure whether the binding to the first (non-vnet) jail address
> happens before or after the ipfilter rules.

- Jamie