From nobody Sat Dec 11 18:12:44 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CAE5018EB0E4; Sat, 11 Dec 2021 18:12:49 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JBG8c6Kd7z4mPQ; Sat, 11 Dec 2021 18:12:48 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 9B9AA3201C1D; Sat, 11 Dec 2021 13:12:47 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Sat, 11 Dec 2021 13:12:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=gljZvF/rHnAAcaSYBGc+EYdHsTE v9hmixX15qLvS1OY=; b=YQXR1Z0J2oma7vVCAOuB4ocqokmieWHCG2s+N9ZwrGk yv26zPYFixPlk4kyt1rok+c6DV80jmHiEo7FupHaxx1ZIX/Xy8Aifmnh3ZnMVGWz iqNn6oZNqNHrxFdb3jC1w5M8Fs225arWUffz8jetKuHw+nx7GwfC2KUpDOu2ANGv 9FHJePu4Gm5CUzEa6IoQXi/KGYD+xGqk9Oal2d/ZNQ5VKl6VWCc56po9vvyteLBe AwKbqbYgdpfVY9sWFz3e9Vce3BQ2HE+tL/q4F7pl5gd3vy4udpZEAU3P4cdQobyU h81HfiOsc8MC/VLOi6hqNBjy18o7IBF65n3w3Q+9+sw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=gljZvF /rHnAAcaSYBGc+EYdHsTEv9hmixX15qLvS1OY=; b=WWloPRL51If7K9jRxyYv3/ ok1Tqy9lUA0iQBoeaUZ6ScjuiUC9cHaq0qkZqZjygxTz4ZGN2SY6WGy5xy9ctqFv 38ykmf7kr4w4Gf9bRi5nzJz8VfFXqbHc+Ke7A+cXh/+EEX/yoE8EQLJKhrd3z7Y7 UpTHNtkUr7xjSLjdyyneb8lg/L6fmMyvz7x1IW1b+BVnmUkS/0Ipefv4rsFJudY7 otlnszeLdmNjsQdA3nV0MBl9KMv6+NPQHh1qSYrEe3Q1DjgS2x/ovKSlbNBjYF+e IPBFRSfUjf2K1aVjSftWtyNBJ57BWLvci94GWsl/G4XxrdQOw73ocnhj/gx6bI9Q == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrkeeggddutdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucggtffrrghtthgvrhhnpefglefhvddvkeehgeeivedtlefhgf euteefhfetvdehueetvdegveefteejtddtheenucffohhmrghinhepfhhrvggvsghsugdr ohhrghdprggrrhgthheigedrihhtnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepthgvtghhqdhlihhsthhsseiihiigshhtrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 11 Dec 2021 13:12:46 -0500 (EST) Date: Sat, 11 Dec 2021 18:12:44 +0000 From: tech-lists To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: pf cannot allocate memory after a time Message-ID: Mail-Followup-To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org References: List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Y//rnzCKUsuPuT8M" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4JBG8c6Kd7z4mPQ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm1 header.b=YQXR1Z0J; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=WWloPRL5; dmarc=none; spf=none (mx1.freebsd.org: domain of tech-lists@zyxst.net has no SPF policy when checking 64.147.123.25) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-3.36 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm1,messagingengine.com:s=fm1]; RWL_MAILSPIKE_POSSIBLE(0.00)[64.147.123.25:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zyxst.net]; NEURAL_SPAM_MEDIUM(0.73)[0.727]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.40)[0.396]; NEURAL_HAM_SHORT(-0.99)[-0.985]; SIGNED_PGP(-2.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.25:from] X-ThisMailContainsUnwantedMimeParts: N --Y//rnzCKUsuPuT8M Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, On Sat, Dec 11, 2021 at 06:04:17PM +0100, Marcel Bischoff wrote: > I've stumbled into this issue some time ago as well. The usual remedies= =20 > of raising states and table-entries did not help. > > I could resolve this with a combination of increasing the process memory= =20 > limit and lowering ZFS memory usage. I'm guessing that you do use ZFS,=20 > since you have an 8 GB Pi, if not please disregard. You're right, I am using zfs > FreeBSD limits the memory available per process to 512 MB by default.=20 > It appears that large PF tables cause issues with this default.=20 > Raising that limit to 2 GB seems to have done the trick. To attempt this,= =20 > add the "kern.maxdsiz" tunable to /boot/loader.conf and reboot, like so: > >kern.maxdsiz=3D"2147483648" Yes, came to this conclusion shortly before you posted this. I've written= =20 what I've done further in https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-memo= ry-since-upgrade-to-13-0.80822/#post-546253 I found this thread https://forums.freebsd.org/threads/pf-states-limit-reached.67337/ informative too. But the idea of increasing it came from=20 https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html I note that on recent -current (main-n251261) on aarch64 kern.maxdsiz=20 seems to be 1073741824 by default. But on amd64 on releng/12.2-p7 and on releng/13.0-p5 and stable/13=20 it defaults to 34359738368 ! On armv6 r366954 12.2 release it's 536870912.=20 I don't know why it's so low on current (arm64.aarch64). It would explain= =20 why i've only seen this issue on this arch. > Sometimes PF still tripped up when replacing one large table with another= =20 > (via pf-badhosts). Usually this happened when the generated list to repla= ce=20 > grew to several MB in size. I've read somewhere that PF needs double the= =20 > memory to replace a table since for an instant it needs to hold both tabl= es=20 > in memory, which makes sense to me. You can verify that this is the case = by=20 > first dropping and then recreating the pf-badhost table with all entries.= =20 > That usually works, while replacing (what pf-badhosts does) usually does = not. Yes, can confirm it's the replacing where the errors arose. > In those cases I found the ZFS ARC cache to be the culprit. By default th= is=20 > grabs most of the available memory to cache file access and frees it on a= pplication=20 > request. Again, PF appears to require the memory quicker than ZFS appears= to be=20 > able to free it. Usually the default ARC settings are fine but in this ca= se you=20 > may want to tune them for testing purposes, again in /boot/loader.conf an= d reboot: > >vfs.zfs.arc_max=3D"2000M" >vfs.zfs.arc_min=3D"500M" Thank you for this, I'll do this if required. For now, I've set kern.maxdsi= z=3D4294967296 and am monitoring the pfbsdhost's mailbox for error reports. --=20 J. --Y//rnzCKUsuPuT8M Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmG06hIACgkQs8o7QhFz NAXklBAAmiKX9PHyF7rcd0H2SU2HRLOrkdNTX9WkQQN1uoPrc1qoQFqPhMttloPU WofRsG44cWMd+JSDC7IW74NyRVNqdOqxhoC9iryuez6IOrR+ruFMTnY/yv103N6/ yNXZqkSu1C5CRlBuq4cihQBDFTo849lRBU3Wjw/IAHGjSgDpHgLNVTXLl1vXrydi YxgkgpEXs75NduK9lGRTTrguicnFGAn1HxxDbEztw3y9pW5pHm9a2oDR7QnofPk8 sYJ+VogZu+xJVzN/EgWcdrSFxInXvxV8lGSeylpOvReg44xiNfdTEKGvGdSLOAWo Y+WwGzS6cuGPSIv1FufqkM4h4Vx0LFWFWAMyleOQPEHCTV6nbGr2/v671VqoilBd 2urK9Xf7YCRrUkG9Gt77974fjJtK5PANQ60dpTCz1bapbEayvk9+mDVwLG8N/QpO EI9keyPclrNd500XzVjFJNOl0+8e5/DC4GhET6WCQuxd1E16A+CB30qMpQOx/Kmq iz/jZv8dtxetyjt1MmfJXd7KCYkrMhLEQR7FfrPJUfHQ1reWDadhy9xYbGvr0HBF CzAKVMyCHtaamUyahn0g7xNb2zdX6CiQWhZZyVwyXV/lu4qk/xaxw34xPGH9xFlW /+LUk1I3KTcvlikZLRPYVKu6TLMYk5fXJTUTG4k+fzx5JfqlSNw= =ZV8m -----END PGP SIGNATURE----- --Y//rnzCKUsuPuT8M--