IPFW and in kernel nat

Sun Aug 30 02:59:24 UTC 2015

I have done some further work on this and I have been able to get kernel nat working in ipfw but some functionality I am after didn’t work.

As I said in the below email I have a couple of jails with public ips, those jails are unable to communicate with the jails on the private network behind nat. The traffic seems to go via the loopback interface, which is then blocked because its not going through nat.

I have also found that while jails can communicate with each other (hairpin nat is working) some of the jails cannot. The jails that cannot communicate with each other are the ones which have ports redirected to them. For instance, I have a jail on 10.1.0.9 and it is unable to communicate with the jail on 10.1.0.2 via port 65432. The traffic seems to be going over the loopback interface and not via nat. Any traffic between jails where there is no redirected ports (via nat) work fine. I have no custom routes, only what the system sets up when adding ips via /etc/rc.conf. It seems this may be a bug with ipfw and kernel nat?

Regards,

Nathan Aherne
nathan at reddog.com.au
+61 402 555 080

> On 22 Aug 2015, at 1:19 pm, Nathan Aherne <nathan at reddog.com.au> wrote:
> 
> Hi Everyone,
> 
> First time message to this list, so I am sorry if I do something against the rules.
> 
> I have posted this questions on the FreeBSD forums in two different places but have not had a single response in several days, so thought I might get more success here. I have spent many days solely reading about IPFW and in kernel nat in the hopes that I would be able to get what I want to achieve working without having to ask for help. I have found it extremely hard to find what I would think is a regular use use case for IPFW and nat. There are examples but there are just so many that are bad or hard to follow, including the handbook and IPFW that its extremely easy to get confused, as I have. Because of this, I will be posting my eventual solution back to the forums in the hopes that it stops someone from wasting days.
> 
> I have a single wan interface (bce0) that has two public IPs attached to it. I would like to use one of these IPs for the host (wanip1) and jails and another for a jail (wanip2) that requires a public IP. I have a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the jails reside on. I would like to forward ports 80,443 coming in on wanip1 to the jail proxy, which then forwards the traffic off to the correct jail. I would like for the host server to be able to have its own firewall and also receive traffic on port 65222. I have a couple of other jails that require some other ports as you will see in my ipfw.rules script. I would also like each jail to be able to connect to the internet.
> 
> I would like the jails to be able to have their own firewalls, which I currently allow by including a firewall script from within each jail.
> 
> I have net.inet.ip.fw.one_pass=0 set in /etc/sysctl.conf
> 
> Below is my ipfw.rules script. I would super appreciate it if someone could show me where I have gone wrong.
> 
> *****************************************************************************************************************************************************************************
> #!/bin/sh
> 
> ######################################################
> # Configuration
> wif="bce0" # WAN interface
> wip="119.111.111.111" # WAN IP
> 
> j1if="lo1" # Jails Interface
> j1net="10.1.0.0/24" # Jails Network
> skip="skipto 30000"
> jcmd="ipfw -q add 10000” # Jails rules are inserted here
> ######################################################
> # IPFW variables
> cmd="ipfw -q add"
> ks="keep-state"
> sks="setup keep-state"
> ######################################################
> ipfw -q -f flush # Flush all rules
> ######################################################
> # NAT on Jail1 WAN IP
> ipfw nat 1 config ip $wip same_ports unreg_only reset \
>                         redirect_port tcp 10.1.0.1:80 80 \
>                         redirect_port tcp 10.1.0.1:443 443 \
>                         redirect_port tcp 10.1.0.2:65432 65432 \
>                         redirect_port tcp 10.1.0.3:65444 65444 \
>                         redirect_port tcp 10.1.0.3:65333 65333
> 
> # Jail1 Network - allow all traffic
> $cmd 10 allow ip from any to any via $j1if
> ######################################################
> # Allow all traffic on Loopback
> $cmd 999 allow ip from any to any via lo0
> 
> # NAT Rule for incoming packets on WAN IP
> $cmd 1000 nat 1 ip4 from any to any in via $wif
> 
> # Check stage table
> $cmd 2000 check-state
> ######################################################
> # HOST ONLY
> # Ping
> $cmd 2100 allow icmp from $wip to any out $ks
> $cmd 2101 allow icmp from any to $wip in $ks
> # DNS
> $cmd 2102 allow tcp from $wip to any 53 out $sks
> $cmd 2103 allow udp from $wip to any 53 out $ks
> # Ports
> $cmd 2104 allow tcp from $wip to any 80 out $ks
> $cmd 2105 allow tcp from $wip to any 433 out $ks
> # SSH
> $cmd 2106 allow tcp from $wip to any 22 out $ks
> $cmd 2107 allow tcp from $wip to any 65222 out $ks
> $cmd 2108 allow tcp from any to $wip 65222 in $ks
> # OpenNTP
> $cmd 2109 allow udp from $wip to any 123 out $ks
> ######################################################
> # Jails   
> # Out
> $cmd 3004 $skip ip4 from any to any out xmit $wif $ks
> # In
> $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks
> $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks
> $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks
> $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks
> $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks
> ######################################################
> 
> # jail.example.com <http://jail.example.com/>
> . /usr/jails/jail.example.com/ipfw.rules <http://jail.example.com/ipfw.rules>
> 
> $cmd 30000 nat 1 ip from $jnet to any out
> 
> ######################################################
> # Deny Remainder and Log
> $cmd deny log all from any to any
> ######################################################
> 
> *****************************************************************************************************************************************************************************
> 
> Regards,
> 
> Nathan
> 