trouble getting Jail with IPFW+NAT to work

Sat Jul 31 17:01:23 UTC 2010

On Sat, 31 Jul 2010, Rick van der Zwet wrote:

 > I like to run Jails on this system [FreeBSD 8.0-RELEASE-p4 (amd64)]
 > and the Jails should be enabled for access to the outside world using
 > NAT as I have only external IP address, The jails are connected to
 > ip's configured on the lo1 interfaces.
 > 
 > ICMP packets seems to flow out and in looking at my tcpdump, but the
 > new got received by my Jail anymore. A natd setup does not work
 > either. if I use the pf firewall how-ever it works like a charm.
 > 
 > Is this setup not supported by IPFW+NAT or am I doing something wrong?

The latter.

 > /Rick
 > 
 > I test my connection using:
 >     # ping -c 1 8.8.8.8 >/dev/null ; echo $?
 >     0
 >     # jls | grep 13
 >       13  10.0.0.2        wleiden.vanderzwet.net        /usr/jail/wleiden
 >     # jexec 13 ping -c 1 10.0.0.1 > /dev/null ; echo $?
 >     0
 >     # jexec 13 ping 8.8.8.8	
 >     ^C
 >     --- 8.8.8.8 ping statistics ---
 >     15 packets transmitted, 0 packets received, 100.0% packet loss
 > 
 > Tcpdump when looking at the last ping:
 >     # tcpdump -i re0 ip proto 1
 >     11:04:33.176393 IP 78.46.85.230 > 8.8.8.8: ICMP echo request, id
 > 43582, seq 313, length 64
 >     11:04:33.183051 IP 8.8.8.8 > 78.46.85.230: ICMP echo reply, id
 > 43582, seq 313, length 64
 >     11:04:34.186391 IP 78.46.85.230 > 8.8.8.8: ICMP echo request, id
 > 43582, seq 314, length 64
 >     11:04:34.192663 IP 8.8.8.8 > 78.46.85.230: ICMP echo reply, id
 > 43582, seq 314, length 64
 > 
 > = /etc/rc.conf relevant snippets =
 > firewall_enable="YES"
 > firewall_nat_enable="YES"
 > firewall_script="/etc/rc.firewall.local"
 > 
 > cloned_interfaces="lo1"
 > ifconfig_lo1="inet 10.0.0.1 netmask 255.255.255.0"
 > ifconfig_lo1_alias0="inet 10.0.0.2 netmask 255.255.255.0"
 > 
 > gateway_enable="YES"
 > 
 > jail_enable="YES"
 > jail_wleiden_rootdir="/usr/jail/wleiden"
 > jail_wleiden_hostname="wleiden.vanderzwet.net"
 > jail_wleiden_ip="10.0.0.2"
 > jail_wleiden_devfs_enable="YES"
 > jail_wleiden_devfs_ruleset="devfsrules_jail"
 > 
 > = relevant sysctl entries =
 > net.inet.ip.forwarding: 1
 > security.jail.allow_raw_sockets: 1
 > net.inet.ip.fw.enable: 1
 > 
 > = /etc/sysctl.conf =
 > security.jail.allow_raw_sockets=1
 > 
 > = Loaded modules =
 > %kldstat
 > Id Refs Address            Size     Name
 >  1   17 0xffffffff80100000 d188c0   kernel
 >  2    1 0xffffffff80e19000 20ab0    geom_mirror.ko
 >  4    1 0xffffffff8102d000 7f2      accf_http.ko
 >  5    1 0xffffffff8102e000 1ea      accf_data.ko
 >  6    1 0xffffffff8102f000 1f3e     nullfs.ko
 >  8    3 0xffffffff81022000 a1d1     ipfw.ko
 >  9    1 0xffffffff81031000 14d5     ipfw_nat.ko
 > 10    1 0xffffffff81033000 b39a     libalias.ko
 > 11    1 0xffffffff8103f000 163f     ipdivert.ko

I'll take all of your jail setup on faith, but ..

 > = /etc/rc.firewall.local =
 > #!/bin/sh -
 > fwcmd="/sbin/ipfw"
 > 
 > ############
 > # Flush out the list before we begin.
 > ${fwcmd} -f flush
 > 
 > ${fwcmd} add 100 pass all from any to any via lo0
 > 
 > # Also tested using the lines below
 > # natd -interface re0 -verbose | tee -i /tmp/natd.log &
 > # ${fwcmd} add divert natd all from 10.0.0.0/24 to any via re0
 > ${fwcmd} add nat 200 all from 10.0.0.0/24 to any via re0
 > ${fwcmd} nat 200 config if re0
 > 
 > ${fwcmd} add 65001 allow all from any to any

.. here you're only doing NAT on the way out, ie packets from 10.x are 
only 'via re0' on the way out - they have no receive interface on the 
way in, being from the local host, see ipfw(8).

But mainly, you have no nat rule for the response packets coming in on 
the outside interface, which is where they need to get mapped back to 
the internal address/es.  Generally better to not use 'via' but be more 
specific (ie clear) about direction on nat rules:

${fwcmd} add nat 200 all from 10.0.0.0/24 to any out xmit re0
${fwcmd} add nat 200 all from any to ${outside_addr} in recv re0

$outside_addr can be 'any', if you're not routing other addresses.

Perhaps also specify ip4 rather than all, if that's what's implied.  
Certainly passing ip6 packets to natd is bad news (panics, currently)

cheers, Ian

 > == pf setup ==
 > 
 > = Loaded modules =
 > %kldstat
 > Id Refs Address            Size     Name
 >  1   11 0xffffffff80100000 d188c0   kernel
 >  2    1 0xffffffff80e19000 20ab0    geom_mirror.ko
 >  4    1 0xffffffff8102d000 7f2      accf_http.ko
 >  5    1 0xffffffff8102e000 1ea      accf_data.ko
 >  6    1 0xffffffff8102f000 1f3e     nullfs.ko
 > 11    1 0xffffffff81031000 2bbc1    pf.ko
 > 
 > = /etc/pf.conf =
 > nat on re0 from lo1:network to any -> (re0)
 > 
 > ## FILTER RULES
 > pass in log all keep state
 > pass out log all keep state
 > 
 > = /etc/rc.conf =
 > pf_enable="YES"
 > 
 > ... [snip: interface/route setup same as above]
 > ... [snip: jail setup same as above]
 > 
 > = Output test =
 > jexec 13 ping -c 3 8.8.8.8
 > PING 8.8.8.8 (8.8.8.8): 56 data bytes
 > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=6.490 ms
 > 64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=6.836 ms
 > 64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=6.252 ms
 > 
 > --- 8.8.8.8 ping statistics ---
 > 3 packets transmitted, 3 packets received, 0.0% packet loss
 > round-trip min/avg/max/stddev = 6.252/6.526/6.836/0.240 ms
 > 
 > 
 > -- 
 > http://rickvanderzwet.nl