Machine freezes when loading pf ruleset

[Kolontai Andrej]

>1.5k rules seems like a lot for PF to handle.
>
>Is that 1.5k rules you've written in the conf, or 1.5k rules from `pfctl -sr | wc -l' ?

Yes, that's what is in the conf files. The latter command gives around 3400...

>I would suggest you find a way to drastically lower that.

Given the number of networks, devices and applications that seems to be difficult. Also, we have some policies on our firewall which contribute to the large number of rules:
* strict minimum principle. Only traffic that is explicitly needed passes the fw. That gives a lot of combinations.
* most rules are either "pass in" or "pass out" . So every connection needs to have a pass in and a pass out rule. This does not mean that we have every rule twice. It's just that inbound and outbound policies are different things and we try not to mix them (accidently). 
* using "quick" is actually not allowed except in very special cases as it tends to cause unexpected side effects. 

What we do is, of course, using tables and anchors to make the evaluation of the rule set as efficient as possible. 

I need to say that in normal operation, the machines perform really well with absolutely no sign of shortage in any resource. 

A typical top screen looks like this:

last pid: 59875;  load averages:  0.29,  0.33,  0.36                                                                                                                                       up 0+02:55:51  15:02:22
38 processes:  1 running, 37 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.7% interrupt, 99.2% idle
Mem: 3588K Active, 1883M Inact, 2333M Wired, 2037M Buf, 89G Free
Swap: 117G Total, 117G Free
...

There's plenty of memory and cpu power. Networking is (physically) capable of 40 gbit/s  (4 aggregated 10 gbit links). I have never seen a cpu load higher than 1.something. 

The problem occurs only in the moment I load the ruleset and only if the machine is in master state. If I only knew what is happening at that point... I can't make any sense of the log outputs. 

>You may also wish to ensure :
>- you're using, if at all possible, a *dedicated* pfsync link (like a cable on a dedicated interface between the boxes)

Yes, to some degree that is true. Pfsync runs over separate physical interfaces. But the machines are at different places. That means that the pfsync net is in fact just another vlan which runs over the same fiber link as everything else. Yes, we used to use IPsec to secure it. Right now, Ipsec is turned off for pfsync to exclude the possibility that this causes the problem. But that's apparently not the case. It still happens.