How to force full sync using pfsync?

[David DeSimone]

David DeSimone <fox at verio.net> wrote:
>
> When I reboot one of the cluster members, the state tables do
> synchronize and populate with some of the same connection states, but
> not all of them.

I still have not figured out why this condition comes about.

> In particular, long-lived, extant connections seem to never show up in
> the rebooted member's state table.

Indeed, it appears that only new connections show up in the state
table on the rebooted cluster member.  It seems to me, though, that
connections which are mostly idle but still receive periodic activity
(such as IRC connections) should eventually find their way into the
state table.  But this does not occur.

> I figured that doing ifconfig down/up would send some sort of "full
> sync" message between the two members, to cause the entire state table
> to be sent in bulk.  But, no such behavior seems to come about.

This part I have figured out.  A bulk update request is only sent when
the "syncdev" option is specified.  So, to perform a full sync update, a
command like this is needed:

    ifconfig pfsync0 syncdev fxp0   # $pfsync_syncdev

When I perform the above command, I see the following debug output (when
PF is configured at "misc" or "loud" debug level):

    On the cluster member receiving the requests:

	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request

    On the cluster member making the request (where syncdev was just
    ifconfig'd):

	pfsync: requesting bulk update
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: failed to receive bulk update status

After performing this manual action, I find the state table is much
better populated, and the two firewalls appear to be synchronized. 
However, the messages above bother me.  It looks to me like the cluster
member making the request repeats it over and over again, and finally
gives up after PFSYNC_MAX_BULKTRIES (12) attempts.  Shouldn't that be
something that only happens in exceptional conditions?  Yet, I can make
it happen every time, even on a test cluster with no traffic (and thus
an almost empty state table).

Does anyone have any insight as to why I see these problems?

1.  Why does pfsync synchronize the state tables when I use the
    "ifconfig syncdev" trick to force a bulk update, yet it does
    not do this when the system is booting up?

2.  Why does pfsync keep repeating the bulk update request and then give
    up?  What message is not getting through?


The two cluster members have a direct cross-cable between them.  My PF
policy has these settings:

    set skip on pfsync0

    pass quick on fxp0 proto pfsync	# $pfsync_syncdev

-- 
David DeSimone == Network Admin == fox at verio.net
  "It took me fifteen years to discover that I had no
   talent for writing, but I couldn't give it up because
   by that time I was too famous.  -- Robert Benchley