Asymmetric routing vs. pf

[Thomas Johnson]

Hello,

I am in the process of trying to build up a new firewall cluster using
FreeBSD 9.2 (-PRERELEASE, r254572) and pf. I am running into some
issues with asymmetric routing, and wondering if there is some piece
of configuration I'm missing/misusing, or if my configuration just
isn't workable (I suspect this).

The firewall cluster consists of two hosts, with identical hardware
configurations. The only difference is that the WAN interface of each
firewall is connected to a different uplink (using BIRD for BGP). The
hosts are connected to each other, running an iBGP session and PFSync.
On the LAN side, I am using CARP to provide internal hosts a gateway.

When I add pf into the mix everything works wonderfully, until I start
testing asymmetric routing situations. I can fabricate a situation
where the inbound leg of the connection comes over one uplink, with
the response going across the other. When I create such a connection
(SSH in this case), I get the following state entries on the two hosts
in the cluster.

On the host receiving the inbound traffic (from the WAN):

root at edge1b:~-> pfctl -vvs states | grep -A 3 :22
all tcp 162.219.166.197:22 <- 192.168.8.1:31467       CLOSED:SYN_SENT
   [0 + 1040]  [1785120388 + 4294964376]
   age 00:00:21, expires in 00:00:12, 17:0 pkts, 3813:0 bytes, rule 1
   id: 5221f22a0000017c creatorid: 754319ef
all tcp 192.168.8.1:31467 -> 162.219.166.197:22       SYN_SENT:CLOSED
   [1785120388 + 4294964376]  [0 + 1040]
   age 00:00:21, expires in 00:00:12, 17:0 pkts, 3813:0 bytes, rule 20
   id: 5221f22a0000017d creatorid: 754319ef

And on the host with the outbound shortest path (is also the LAN CARP master):

root at edge1a:~-> pfctl -vvs states | grep -A 3 :22
all tcp 162.219.166.197:22 <- 192.168.8.1:31467       SYN_SENT:ESTABLISHED
   [983303631 + 61855] wscale 6  [1785117466 + 69482] wscale 6
   age 382765:14:04, expires in 00:00:10, 0:20 pkts, 0:4729 bytes
   id: 5221f22a0000017c creatorid: 754319ef
all tcp 192.168.8.1:31467 -> 162.219.166.197:22       ESTABLISHED:SYN_SENT
   [1785117466 + 69482] wscale 6  [983303631 + 61855] wscale 6
   age 382765:14:04, expires in 00:00:10, 0:20 pkts, 0:4729 bytes
   id: 5221f22a0000017d creatorid: 754319ef

As expected, the connection works until the timers expire on the
half-open states. I was hoping that PF/PFSync would be smart enough to
match up the connection and "do the right thing," but that doesn't
seem to be the case.

I've tried using synproxy state and sloppy state, but neither seem to
work (more likely I'm not using them right), and the reading I've done
suggests that both have serious drawbacks. My pf configuration is as
follows (the relevant bits):

wan_if="em0"
lan_if="em1"
xo_if="em2"

vpn1="162.219.166.197"

table <self>            {self}

# Section 2: Options #

# RST blocked connections
set block-policy drop
# We don't care about OS fingerprinting
set fingerprints "/dev/null"
# Increase state table sizes. Defaults are too small.
set limit { states 200000, frags 20000, src-nodes 20000 }
# Skip pf processing on lo0, just make sure that the default policy
# for inbound to <self> is block
set skip on {$xo_if lo0}

# Section 3: Traffic Normalization #

scrub in all

# Section 6: Policy #

## Default policy
block log all

## "Ingress" traffic handling.
pass in all

## Outbound connection handling
pass out on {$wan_if} modulate state

# Allow SSH traffic to vpn1
# Could maybe be made to work, though OpenBSD doesn't recommend for routine use.
#pass out log on $lan_if proto tcp from any to $vpn1 port 22 synproxy state
# Does not fix state issue.
#pass out log on $lan_if proto tcp from any to $vpn1 port 22 keep state (sloppy)
pass out log on $lan_if proto tcp from any to $vpn1 port 22

#### END OF pf.conf ####

Is the *best* solution here to separate the roles of firewalling and
routing? I have an extra set of hosts that could become firewalls,
making these hosts strictly routers. Would OpenBSD fare any better in
this scenario (thinking specifically of the pfsync "defer"
functionality)?

Thanks!

-- 
Thomas Johnson
ClaimLynx, Inc.

-- 
This e-mail and any files transmitted with it are confidential and are 
intended solely for the use of the individual or entity to whom they are 
addressed. If you are not the intended recipient or the individual 
responsible for delivering the e-mail to the intended recipient, please be 
advised that you have received this e-mail in error and that any use, 
dissemination, forwarding, printing, or copying of this e-mail is strictly 
prohibited.

If you have received this communication in error, please return it to the 
sender immediately and delete the original message and any copy of it from 
your computer system. If you have any questions concerning this message, 
please contact the sender or call ClaimLynx at (952) 593-5969.