From nobody Sat Dec 11 17:04:17 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 84D0118DCD3E; Sat, 11 Dec 2021 17:04:27 +0000 (UTC) (envelope-from marcel@herrbischoff.com) Received: from mailpod.herrbischoff.com (mailpod.herrbischoff.com [IPv6:2a01:4f8:c010:37ae::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mailpod.herrbischoff.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JBDdl17qNz4Zhs; Sat, 11 Dec 2021 17:04:26 +0000 (UTC) (envelope-from marcel@herrbischoff.com) Received: from mailpod.herrbischoff.com (localhost [127.0.0.1]) by mailpod.herrbischoff.com (OpenSMTPD) with ESMTP id 7feb2ea0; Sat, 11 Dec 2021 18:04:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=herrbischoff.com; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=hrbf; bh= qNTupHgznxgk/sQlTiTH5Z21O1vMwSryyCrcwIOc2RI=; b=hJ22XcQ0wCabu4jg gGXHB+duHjm2GhW3t04E6ogU/00mvIGZjWXKLF/ruihNpGGwYTnwNyeruDbYiKzG 4l++Rr0H8o3XivU1J0tFzGIjBKQ00PtWXyJAr6DIKel1qeqjm0mIzUJXPoLDLCMG NQD2Xle4HM30kNXQySWnbamxixRmlWfR/f0dhmad0hd/1ADpjEDJUwxynavB/ZGm poOuOP5dwyo67Xiz/Lba1jihs2KE15kmdMxH7h69qCZgI+8VCKptkEB4oHaJJ6+v ibaKa7ld8O/6Qb28KMTWX1JyAmf3ecVxoDdbJ2Z24+q1YoJ9kgIE6NxkF+urOPX4 ypImlA== DomainKey-Signature: a=rsa-sha1; c=nofws; d=herrbischoff.com; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; q=dns; s= hrbf; b=gSSST4PuvxyiDWIDgglW/ZebBwgeAuFm38CsdM64s5cOCek4xeHTaBZl LJigvxsXWv6JOP3G7bI+LDLhEbGmbro3QJNyTLfG7+FJR3ayJ6fWtYQ0CDLC0mU4 xbZrShI/Ep/KzzjfIKmgFm8/OerUJmaoaHe1ww1+fpOvbkO9Lt4fPVitBwsrqlY3 GeMD6MClGYWKoy9G9rWTLj0RkTr/OfO2lXeLdybTyr20FFXF2mgXAC8q46LyK4cE yqWjSLkWH1hcKikbhpEa0TtGdiI5PRGqrpRuYNMC4Ao6c57poDA6pEXqpH9FaaW4 0FNF6vGpx+xx2plZL9dtfUMcKSPTfA== Received: by mailpod.herrbischoff.com (OpenSMTPD) with ESMTPSA id 867d936f (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) auth=yes user=marcel@herrbischoff.com; Sat, 11 Dec 2021 18:04:18 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: pf cannot allocate memory after a time From: Marcel Bischoff In-Reply-To: Date: Sat, 11 Dec 2021 18:04:17 +0100 Cc: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: tech-lists X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JBDdl17qNz4Zhs X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N I've stumbled into this issue some time ago as well. The usual remedies = of raising states and table-entries did not help. I could resolve this with a combination of increasing the process memory = limit and lowering ZFS memory usage. I'm guessing that you do use ZFS, = since you have an 8 GB Pi, if not please disregard. FreeBSD limits the memory available per process to 512 MB by default. It = appears that large PF tables cause issues with this default. Raising = that limit to 2 GB seems to have done the trick. To attempt this, add = the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so: kern.maxdsiz=3D"2147483648" You can verify this value has taken by examining the output of "ulimit = -d" before and after. See "man tuning" for reference. Sometimes PF still tripped up when replacing one large table with = another (via pf-badhosts). Usually this happened when the generated list = to replace grew to several MB in size. I've read somewhere that PF needs = double the memory to replace a table since for an instant it needs to = hold both tables in memory, which makes sense to me. You can verify that = this is the case by first dropping and then recreating the pf-badhost = table with all entries. That usually works, while replacing (what = pf-badhosts does) usually does not. In those cases I found the ZFS ARC cache to be the culprit. By default = this grabs most of the available memory to cache file access and frees = it on application request. Again, PF appears to require the memory = quicker than ZFS appears to be able to free it. Usually the default ARC = settings are fine but in this case you may want to tune them for testing = purposes, again in /boot/loader.conf and reboot: vfs.zfs.arc_max=3D"2000M" vfs.zfs.arc_min=3D"500M" See https://docs.freebsd.org/en/books/handbook/zfs/#zfs-advanced for = reference. These are just the observations I've made and the solutions I've found. = My reasoning may be sound or completely off. Best, Marcel > On 11. Dec 2021, at 17:14, tech-lists wrote: >=20 > Hi, >=20 > context: main-n251261-25d0ccbe101 on arm64.aarch64 (raspberry = pi4b/8GB) >=20 > I'm trying to use pf with pf-badhosts = (https://geoghegan.ca/pub/pf-badhost/latest/install/freebsd.txt) and am = seeing what *seems like* a pf problem which has been reported elsewhere = in different contexts ( e.g. = https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-mem= ory-since-upgrade-to-13-0.80822/ ) >=20 > from pfctl -sa: > [...] > LIMITS: > states hard limit 100000 > src-nodes hard limit 10000 > frags hard limit 5000 > table-entries hard limit 25400000 [*] > [...] >=20 > [*] the pf-badhosts guide quotes 400000 for this value; I bumped it to = 25400000 in order to "give pf more memory" >=20 > The problem is that if pf tables either get reloaded or if the machine = is running for say over 24 hrs, pf throws errors. This works if the = machine is rebooted but pf isn't switched on: >=20 > [...] > # doas -u _pfbadhost pf-badhost -O freebsd = Password: >=20 > pf-badhost 1512 - - Using experimental "aggy" aggregator... >=20 > 6105 addresses added. > 6235 addresses deleted. >=20 > pf-badhost 1580 - - > IPv4 addresses in table: 619200750 > [...] >=20 > running pfctl -e -f /etc/pf.conf loads and runs. A day or so later, = I'll see this in the logs, after pf-badhost runs its update: >=20 > [...] > pf-badhost 15202 - - Using experimental "aggy" aggregator... >=20 > pfctl: Cannot allocate memory. >=20 > pf-badhost 15256 - - ERROR: '/etc/pf-badhost.txt' contains invalid = data! Reverting changes and bailing out... > [...] >=20 > There's plenty of memory. I've tried running this with one term on top = -P open and there's always 1-2GB available (free) as well as 12GB of = swap which is unused. >=20 > If I try pfctl -Fa -f /etc/pf.conf and log back in and then run = pf-badhost manually: >=20 > [...] > # doas -u _pfbadhost pf-badhost -O freebsd [...] >=20 > not only the pfbadhost table doesn't load but nothing loads: >=20 > [...] > # pfctl -e -f /etc/pf.conf > /etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory > /etc/pf.conf:23: cannot define table rfc6890: Cannot allocate memory > /etc/pf.conf:26: cannot define table gooDNS6: Cannot allocate memory > /etc/pf.conf:27: cannot define table friends: Cannot allocate memory > pfctl: Syntax error in config file: pf rules not loaded > [...] >=20 > The only solution is a reboot. How to fix? Do I need to increase = src-nodes/frags? >=20 > thanks, > --=20 > J.