Kernel NAT issues

Wed Nov 18 00:48:22 UTC 2015

Hi Everyone,

I think I have worked this out and have a working Stateful IPFW NAT (its worked fine for a few weeks) rule set. Hopefully this saves someone else a few weeks of their lives.

For some reason hairpin (loopback nat or nat reflection) does not seem to be working, which is why I chose IPFW in the first place.

I am actually getting some really weird log results from IPFW (for inter jail communication ). IPFW logs show traffic flowing in the opposite direction to what tshark shows. Tshark shows the traffic in the direction I would expect. IPFW is blocking the traffic because it is seeing it as going the wrong way. Anyone have an idea how I can solve this issue, I would super appreciate any pointers!

To clarify inter jail communication - not using the local jail IPs but DNS - they all resolve to the same public IP. I cannot use split DNS.

#######################################################################################################################################################
#!/bin/sh

# Configuration
wif="bce0" # WAN interface
jif="lo1" # Jail interface
jsn="10.0.0.0/16" # Jail subnet
plip="10.0.0.1" # Proxy IP
hwip=“aaa.bbb.ccc.ddd" # Host WAN IP
nwip=“www.xxx.yyy.zzz" # NAT WAN IP

# Script variables
cmd="ipfw -q add"
ks="keep-state"
sks="setup keep-state"

# Flush all rules
ipfw -q -f flush

# Enable NAT
ipfw nat 1 config ip $nwip log

# Allow all loopback traffic
$cmd 5 allow ip from any to any via lo0

# Block any traffic not for Host
$cmd 6 deny ip from any to not me in via $wif

$cmd 100 nat 1 log ip from any to $nwip recv $wif
$cmd 101 check-state log

# Host
$cmd 110 allow icmp from any to $hwip in via $wif $ks
$cmd 111 allow tcp from any to $hwip 65222 in via $wif $sks
$cmd 112 allow icmp from $hwip to any out via $wif $ks
$cmd 113 allow tcp from $hwip to any 53, 80, 443, 22, 65222 out via $wif $sks
$cmd 114 allow udp from $hwip to any 53, 123 out via $wif $ks

# Incoming NAT
$cmd 120 skipto 65510 log tcp from any to $jsn recv $wif $sks
$cmd 121 skipto 65510 log udp from any to $jsn recv $wif $ks
# Outgoing NAT
$cmd 122 skipto 65510 log tcp from $jsn to not $jsn xmit $wif $sks
$cmd 123 skipto 65510 log udp from $jsn to not $jsn xmit $wif $ks

# JAILS RULES 200-65000

# Block any other traffic
$cmd 65501 deny log ip from any to any

# Outgoing NAT
$cmd 65510 nat 1 log ip from $jsn to any xmit $wif
$cmd 65511 allow log ip from $nwip to any xmit $wif
# Block any other traffic
#$cmd 65519 deny log ip from any to any

# Incoming NAT
$cmd 65520 allow log ip from any to $jsn recv $wif
# Block any other traffic
$cmd 65522 deny log ip from any to any

#######################################################################################################################################################

Regards,

Nathan

> On 21 Oct 2015, at 9:02 am, Nathan Aherne <nathan at reddog.com.au> wrote:
> 
> Hi Ian,
> 
> Thank you very much for your response! Sorry about the late response, I have been offline for a few days.
> 
> I think I may have worked this issue out. I am bringing up a bunch of Jails today to test my firewall rules in the hopes that I have corrected my problem. I will reply back either way.
> 
> Regards,
> 
> Nathan
> 
>> On 15 Oct 2015, at 12:51 am, Ian Smith <smithi at nimnet.asn.au> wrote:
>> 
>> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>>> Hi Ian,
>>> 
>>> Thank you for your response.
>>> 
>>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>>> incorrect.
>>> 
>>> I have now included my ruleset below (as well as my initial email).
>> 
>> Hi Nathan,
>> 
>> I was really hoping someone who knows more about stateful rule handling 
>> (and jail networking) might have a go at this.  Oh well I'll try, but 
>> I'm a lousy mindreader, and really don't know which of the below 
>> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
>> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
>> 
>>> # Enable NAT
>>> ipfw nat 1 config ip $jip same_ports log
>> 
>> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
>> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
>> to you, and so traffic to it won't be subject to NAT .. correct?  But 
>> the WWW... address and all 10.0/16 addresses are jails, not any separate 
>> boxes you gateway for, right?  Just the one external interface, right?
>> 
>>> 00005 allow ip from any to any via lo0
>>> 00006 deny ip from any to not me in via bce0
>>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
>>> 00101 check-state
>> 
>> Ok, inbound from WAN is nat'd and existing stateful flows followed by 
>> executing the rule that originally kept state.  Where this is a skipto, 
>> skipto will be performed.  But where it's a nat rule, I've no idea .. 
>> see below, but you really don't want to add keep-state (again) there.
>> 
>>> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 keep-state
>> 
>> Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
>> rule would make more sense especially for inbound ICMP.  But moving on ..
>> 
>>> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 65222 recv bce0 setup keep-state
>> 
>> Ok, but showting why plain text works better than HTML on lists :)
>> 
>>> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 keep-state
>>> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state
>>> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,123 xmit bce0 keep-state
>> 
>> Smells ok.
>> 
>>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
>>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
>> 
>> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
>> got a problem .. see below.  These two are inbound traffic (recv) and as 
>> is, skipping to 65501 will fall through two outbound rules to be denied.
>> 
>> Either allow them here directly, or likely better, skipto a separate
>> target that then allows (or denies) them, if that's what you intended?
>> 
>>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state
>>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
>> 
>> Ok, this traffic does needs to be NAT'd on the way out.
>> 
>>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
>>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
>>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
>> 
>> Not clear why these tcp ports are open inbound and outbound?  Presumably 
>> this is jail-to-jail traffic?  Perhaps not relevant to your problem.
>> 
>>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
>>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
>>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
>>> 65500 deny log ip from any to any
>> 
>> Ok.
>> 
>>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
>> 
>> This the target for outbound traffix, xmit bce0, so nat is appropriate.  
>> Does jail-to-jail traffic travels via lo1?  Or what?
>> 
>> This won't do anything to inbound traffic, but that really shouldn't get 
>> here except returns as the result of check-state - not from 120 & 121.
>> 
>> But keep-state is not ok, state was already set on the skipto.  I don't 
>> know how this extra keep-state might behave - does anyone have an idea?
>> 
>> Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic.
>> 
>>> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
>> 
>> So, only remaining traffic is outbound from the host itself, and traffic 
>> that is to 10.0/16, but not from AAA... is to be dropped, correct?
>> 
>> I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp 
>> states .. myself, I'd go for separate rules for each eg tcp, udp, .. and 
>> I'd do it somewhere else than as a fall through from outbound nat rule, 
>> it's confusing here, to me anyway .. unless I've missed the reason?
>> 
>>> 65534 deny log ip from any to any
>>> 65535 deny ip from any to any
>> 
>> 
>> Ok, now for your demo of the problem from the later mail, which I've 
>> reformated to quote properly, so:
>> 
>>> To further illustrate my issue, this is a small log output.
>>> 
>>> I am running host google.com <http://google.com/> in the jail, which 
>>> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state 
>>> rule.
>> 
>> I see you don't have logging on 101 above now.  Probably best.
>> 
>>> I would expect the first piece of traffic out would be UNKNOWN 
>>> (does not have an entry in the state table) but it seems the 
>>> returning traffic is also showing as UNKNOWN (the second 101).
>> 
>> I've never logged a check-state, but UNKNOWN may not mean that ..
>> 
>>> You can see that the traffic is returning on the same port it went 
>>> out on, so its obviously the returning traffic. I am not sure why 
>>> state is not being kept?
>> 
>> Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, so 
>> it's been correctly NAT'd on the way in.  Get rid of that keep-state on 
>> the nat rule at 65501 and see if not creating double entries in the 
>> state table helps.  And change the skipto target on 120 & 121 to only 
>> pass outbound traffic to outbound NAT rule/s.
>> 
>> Once you've done outbound NAT, probably best just to 'allow [log] all'?
>> 
>>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>>> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
>>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
>>> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
>> 
>> That said, I can see why this return packet would be denied even if it 
>> were in the nat table: it would execute 'skipto 65501', which nat rule 
>> does not apply, as it's not outbound, and rule 65502 does not apply, as 
>> it's neither from AAA... nor outbound, so it's then denied by 65534.
>> 
>> Hope this helps.  Please cc me on any response to the list.
>> 
>> It would be great if someone else might care to lend an oar here; I'm 
>> paddling out of my depth.
>> 
>> cheers, Ian
>> 
>> [..]
> 
> _______________________________________________
> freebsd-ipfw at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe at freebsd.org"