setfib (ez)jails and wierd routing

[Marko Cupać]

Hi,

I have already asked this on -jail two weeks ago, but perhaps this is
better place to ask.

I notice wierd routing in my setfib (ez)jails setup.

I have a server with multiple NICs. setfib should ensure that LAN jails
(setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but
need to go through firewalls as though they were physical boxes.

pacija at warden3:~ % sudo setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
10.30.19.160/27    00:1c:c4:de:0a:86  US         bce0
127.0.0.1          lo0                UHS         lo0
127.0.1.0/24       lo1                US          lo1

pacija at warden3:~ % sudo setfib 2 netstat -rn
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            193.53.106.254     UGS        bce1
127.0.0.1          lo0                UHS         lo0
127.0.2.0/24       lo2                US          lo2
193.53.106.0/24    00:1c:c4:de:0a:84  US         bce1

Host has the same default route as fib 1:

pacija at warden3:~ % sudo netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
...

If I ssh from the Internet into DMZ jail, everything works as expected.
But if I ping DMZ jail from the Internet, I see reply packets leaving
not the interface they came from (bce1, public address space, DMZ), but
another one (bce0, private address space, LAN). This is kinda
understandable, because jail on fib2 does not have ICMP enabled, so
it is not DMZ jail, but the host (which is in fib 0) who replies to
packets via its default gateway (router on a private LAN).

Is there an easy and elegant way to solve this? Like binding IP address
to fib? I wouldn't like to have to fire up pf on host and meddle with
reply-to rules in order to achieve this, I'd rather revert to old setup
of separate physical servers for each network.

Thank you in advance,

-- 
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/