Re: pf cannot allocate memory after a time

Hi,

On Sat, Dec 11, 2021 at 06:04:17PM +0100, Marcel Bischoff wrote:
> I've stumbled into this issue some time ago as well. The usual remedies 
> of raising states and table-entries did not help.
>
> I could resolve this with a combination of increasing the process memory 
> limit and lowering ZFS memory usage. I'm guessing that you do use ZFS, 
> since you have an 8 GB Pi, if not please disregard.

You're right, I am using zfs

> FreeBSD limits the memory available per process to 512 MB by default. 
> It appears that large PF tables cause issues with this default. 
> Raising that limit to 2 GB seems to have done the trick. To attempt this, 
> add the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so:
>
>kern.maxdsiz="2147483648"

Yes, came to this conclusion shortly before you posted this. I've written 
what I've done further in
https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-memory-since-upgrade-to-13-0.80822/#post-546253

I found this thread
https://forums.freebsd.org/threads/pf-states-limit-reached.67337/
informative too. But the idea of increasing it came from 
https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html

I note that on recent -current (main-n251261) on aarch64 kern.maxdsiz 
seems to be 1073741824 by default.

But on amd64 on releng/12.2-p7 and on releng/13.0-p5 and stable/13 
it defaults to 34359738368 !

On armv6 r366954 12.2 release it's 536870912. 

I don't know why it's so low on current (arm64.aarch64). It would explain 
why i've only seen this issue on this arch.

> Sometimes PF still tripped up when replacing one large table with another 
> (via pf-badhosts). Usually this happened when the generated list to replace 
> grew to several MB in size. I've read somewhere that PF needs double the 
> memory to replace a table since for an instant it needs to hold both tables 
> in memory, which makes sense to me. You can verify that this is the case by 
> first dropping and then recreating the pf-badhost table with all entries. 
> That usually works, while replacing (what pf-badhosts does) usually does not.

Yes, can confirm it's the replacing where the errors arose.

> In those cases I found the ZFS ARC cache to be the culprit. By default this 
> grabs most of the available memory to cache file access and frees it on application 
> request. Again, PF appears to require the memory quicker than ZFS appears to be 
> able to free it. Usually the default ARC settings are fine but in this case you 
> may want to tune them for testing purposes, again in /boot/loader.conf and reboot:
>
>vfs.zfs.arc_max="2000M"
>vfs.zfs.arc_min="500M"

Thank you for this, I'll do this if required. For now, I've set kern.maxdsiz=4294967296
and am monitoring the pfbsdhost's mailbox for error reports.

-- 
J.