Re: pf cannot allocate memory after a time

I've stumbled into this issue some time ago as well. The usual remedies of raising states and table-entries did not help.

I could resolve this with a combination of increasing the process memory limit and lowering ZFS memory usage. I'm guessing that you do use ZFS, since you have an 8 GB Pi, if not please disregard.

FreeBSD limits the memory available per process to 512 MB by default. It appears that large PF tables cause issues with this default. Raising that limit to 2 GB seems to have done the trick. To attempt this, add the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so:

kern.maxdsiz="2147483648"

You can verify this value has taken by examining the output of "ulimit -d" before and after. See "man tuning" for reference.

Sometimes PF still tripped up when replacing one large table with another (via pf-badhosts). Usually this happened when the generated list to replace grew to several MB in size. I've read somewhere that PF needs double the memory to replace a table since for an instant it needs to hold both tables in memory, which makes sense to me. You can verify that this is the case by first dropping and then recreating the pf-badhost table with all entries. That usually works, while replacing (what pf-badhosts does) usually does not.

In those cases I found the ZFS ARC cache to be the culprit. By default this grabs most of the available memory to cache file access and frees it on application request. Again, PF appears to require the memory quicker than ZFS appears to be able to free it. Usually the default ARC settings are fine but in this case you may want to tune them for testing purposes, again in /boot/loader.conf and reboot:

vfs.zfs.arc_max="2000M"
vfs.zfs.arc_min="500M"

See https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced for reference.

These are just the observations I've made and the solutions I've found. My reasoning may be sound or completely off.

Best, Marcel

> On 11. Dec 2021, at 17:14, tech-lists <tech-lists@zyxst.net> wrote:
> 
> Hi,
> 
> context: main-n251261-25d0ccbe101 on arm64.aarch64 (raspberry pi4b/8GB)
> 
> I'm trying to use pf with pf-badhosts (https://geoghegan.ca/pub/pf-badhost/latest/install/freebsd.txt) and am seeing what *seems like* a pf problem which has been reported elsewhere in different contexts ( e.g. https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-memory-since-upgrade-to-13-0.80822/ )
> 
> from pfctl -sa:
> [...]
> LIMITS:
> states        hard limit   100000
> src-nodes     hard limit    10000
> frags         hard limit     5000
> table-entries hard limit 25400000 [*]
> [...]
> 
> [*] the pf-badhosts guide quotes 400000 for this value; I bumped it to 25400000 in order to "give pf more memory"
> 
> The problem is that if pf tables either get reloaded or if the machine is running for say over 24 hrs, pf throws errors. This works if the machine is rebooted but pf isn't switched on:
> 
> [...]
> # doas -u _pfbadhost pf-badhost -O freebsd                                                                         Password:
> 
> pf-badhost 1512 - - Using experimental "aggy" aggregator...
> 
> 6105 addresses added.
> 6235 addresses deleted.
> 
> pf-badhost 1580 - -
> IPv4 addresses in table:  619200750
> [...]
> 
> running pfctl -e -f /etc/pf.conf loads and runs. A day or so later, I'll see this in the logs, after pf-badhost runs its update:
> 
> [...]
> pf-badhost 15202 - - Using experimental "aggy" aggregator...
> 
> pfctl: Cannot allocate memory.
> 
> pf-badhost 15256 - - ERROR: '/etc/pf-badhost.txt' contains invalid data! Reverting changes and bailing out...
> [...]
> 
> There's plenty of memory. I've tried running this with one term on top -P open and there's always 1-2GB available (free) as well as 12GB of swap which is unused.
> 
> If I try pfctl -Fa -f /etc/pf.conf and log back in and then run pf-badhost manually:
> 
> [...]
> # doas -u _pfbadhost pf-badhost -O freebsd [...]
> 
> not only the pfbadhost table doesn't load but nothing loads:
> 
> [...]
> # pfctl -e -f /etc/pf.conf
> /etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory
> /etc/pf.conf:23: cannot define table rfc6890: Cannot allocate memory
> /etc/pf.conf:26: cannot define table gooDNS6: Cannot allocate memory
> /etc/pf.conf:27: cannot define table friends: Cannot allocate memory
> pfctl: Syntax error in config file: pf rules not loaded
> [...]
> 
> The only solution is a reboot. How to fix? Do I need to increase src-nodes/frags?
> 
> thanks,
> -- 
> J.