Can pfsync be used over router or WAN?

[Bruce Simpson]

Sam Wun wrote:
> Establish a IPSEC bewteen this 2 pfsync points is a way to go.
>   

Yup. The key observation about pfsync is that you can configure the 
peer(s) for synchronization in the 'syncdev' mode or the 'syncpeer' mode.
Unlike CARP, pfsync(4) has no authentication built-in.

With syncdev, you are telling pfsync to periodically send out state 
updates to a link-scope IPv4 multicast group.

Obviously, this only works if all the peer(s) are on-link (i.e. the same 
LAN), and any Layer 2 switches in the middle are configured to forward 
the multicast traffic. The IGMP code will send a membership report for 
the 224.0.0.240 address, unless it's configured explicitly to not do so 
for 224.0.0.0/24 link-scope groups via sysctl, which should appease 
snooping switches. Note that it defaults to IGMPv3 in HEAD, it should 
downgrade to v2 or v1 if it sees a v2 Query.
This mechanism operates wholly independently of CARP.
You can IPSEC encapsulate multicast traffic, but of course that gives 
rise to 'interesting' key distribution problems.

With syncpeer, you are telling pfsync to periodically send out state 
updates to a *single* peer, not a list, and all such traffic is unicasted.

As far as I know, you can't specify multiple peers, so you are limited 
to 1 other member (unless the peer address is a CARP address, or 
anycasted by some other mechanism). This should work just fine with 
IPSEC, provided your key distribution is taken care of.

If your WAN link can carry multicast traffic without additional 
encapsulation (most can, even if they're not link-layer 
multicast-capable), then using 'syncdev' should work fine, although the 
IGMP and MLD code in HEAD will suppress sending membership reports on 
interfaces without the IFF_MULTICAST flag. This doesn't disallow the 
stack from sending multicast traffic, though.

[This should perhaps be revisited, because I can think of situations 
where the WAN link may not have a native link-layer multicast 
capability, but it's still desirable for the IGMP/MLD reports to go 
upstream, i.e. DSL in ATM native mode. Userland PPP via tun(4) needs to 
be told to enable IFF_MULTICAST with the TUNSIFMODE ioctl].

For those who are interested in experiments:
    pfsync(4) could in theory be enhanced to use Source-Specific 
Multicast (SSM) for pushing pf state to multiple border firewalls inside 
an AS boundary -- but it would require knowing all the addresses of the 
peer(s), and you'd be dependent on a multicast routing protocol like PIM 
at a minimum for distributing the traffic throughout your AS, as well as 
needing a unicast routing IGP for the traffic to pass the uRPF checks. 
It would be desirable to use a different address for this than 224.0.0.240.
    You could probably get away with Any-Source Multicast (ASM) for 
distributing the pfsync updates, but I'd advise against that, as ASM is 
a little bit harder to secure -- you don't/can't control the endpoints 
without explicit firewall rules, and of course that introduces recursion 
(you're having to firewall your firewall updates...)

    For kernel hacking: The KPIs involved require that kernel consumers 
do their own SSM housekeeping, though -- splicing of consumer layer 
memberships is only done for sockets, and you'd have to craft your own 
RB-trees, although the multicast code takes care of knitting together 
the right state-change reports to send upstream, doing filter matches 
etc -- that's a different matter. It's for this reason that SSM apps are 
generally best written in userland. Doing SSM in-kernel is possible, 
sure, but the whole point of using a socket for it is that a load of 
stuff gets taken care of for you, and using a socket in-kernel is still 
irksome.

Obviously the more mechanisms you introduce to push out the updates, the 
wider the range of possible points of failure you introduce. pfsync is 
cool because it's a tightly integrated solution to a common problem in 
its space, but it may not be the right choice for all folks in its 
present state.

... By the way, does anyone out there have patches to get pfsync(4) to 
work over IPv6?

cheers,
BMS