Question about pf, NAT and routing. (attempt #2)

[Josh Endries]

For some reason my other message never made it through, so here we are 
again!

A while ago I needed to give a jail access to two networks/IPs (one 
external for a web site and one internal for a DB connection). I ended 
up using a localhost IP (127.0.0.2) for the jail and nat/binat in pf to 
control where the traffic went, depending on destination. I'm trying to 
set up a jail now that's similar. My host has multiple interfaces on 
multiple networks, and the jail is on lo0, and I would like to NAT 
traffic to internal networks from one IP out one interface, and 
everything else out another IP through the external interface.

I found an email on here from jpaetzel (o/) explaining how to use 
route-to, and that works; it fixed default route problem (thanks!). 
Unfortunately that only seems to work if the jail is using an IP on one 
of the interfaces in question. I suppose there is some sort of problem 
between the NATing and routing.

Here is my crazy config:

ra# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
1500 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
         ether 00:1e:0b:ed:f9:ec
         media: Ethernet autoselect (1000baseTX <full-duplex>)
         status: active
         lagg: laggdev lagg0
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
         ether 00:1e:0b:ed:f9:ec
         media: Ethernet autoselect (none)
         status: no carrier
         lagg: laggdev lagg0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
         inet6 ::1 prefixlen 128
         inet 127.0.0.1 netmask 0xff000000
         inet 127.0.0.5 netmask 0xffffffff
         inet 127.0.0.4 netmask 0xffffffff
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
         ether 00:1e:0b:ed:f9:ec
         media: Ethernet autoselect
         status: active
         laggproto lacp
         laggport: bce1 flags=20<DISABLED>
         laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
         options=3<RXCSUM,TXCSUM>
         ether 00:1e:0b:ed:f9:ec
         inet 10.1.0.2 netmask 0xffffff00 broadcast 10.1.0.255
         media: Ethernet autoselect
         status: active
         vlan: 2 parent interface: lagg0
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
         options=3<RXCSUM,TXCSUM>
         ether 00:1e:0b:ed:f9:ec
         inet 192.168.0.91 netmask 0xffffff00 broadcast 192.168.0.255
         media: Ethernet autoselect
         status: active
         vlan: 1 parent interface: lagg0
ra# pfctl -s nat
No ALTQ support in kernel
ALTQ related functions disabled
binat on vlan2 inet from 127.0.0.5 to any -> 10.1.0.2
binat on vlan1 inet from 127.0.0.4 to any -> 192.168.0.91
ra# pfctl -s rules
No ALTQ support in kernel
ALTQ related functions disabled
scrub in all fragment reassemble
block drop all
pass out route-to (vlan2 10.1.0.1) inet from 10.1.0.2 to ! 10.1.0.0/24 
flags S/SA keep state
pass out route-to (vlan1 192.168.0.1) inet from 192.168.0.91 to ! 
192.168.0.0/24 flags S/SA keep state
ra# jls
    JID  IP Address      Hostname                      Path
     22  127.0.0.5       dns                           /jails/dns/root
     21  127.0.0.4       mysql                         /jails/mysql/root

The problem is the same. The jail that has the default route for it's 
NAT (dns) works fine, but the other jail (mysql) doesn't. I get the 
error "Can't assign requested address". The packets from the mysql jail 
don't show up anywhere in tcpdump, packets from the dns jail shows up as 
normal.

:D

Josh