From nobody Thu Nov 04 13:00:17 2021 X-Original-To: pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5936C183D65A for ; Thu, 4 Nov 2021 13:00:29 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HlNzJ4vZxz4mYY for ; Thu, 4 Nov 2021 13:00:28 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: by mail-ed1-x535.google.com with SMTP id g14so21028305edz.2 for ; Thu, 04 Nov 2021 06:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuxpowered-net.20210112.gappssmtp.com; s=20210112; h=to:references:from:subject:message-id:date:user-agent:mime-version :in-reply-to; bh=Wr3czjD8C+TqABEdjmtS88VIcs0aGhY8SkwUYq+/tr0=; b=bDagGbQVUiLv4eb84sH18r64VuXDdbUsuQPjmPhsm555DrLmgrCE8efXsBT3BhNHUd +r7h1xIFKmhjBLuRO9LdmzmUtN7XN4qTjx4xClVQybFYMNdOK87AeCNMzHW2y880my5c nqBvOz9aEAwPP6RcWlwabkdco3RFsp5Z+0T/CXKuaY/2svMzd43BYL7isoETWYckGTbH CUwGMLByWmW10ZvhzVUIcj4PaoPlhN+ovcKbf4XRLl7tycftuppzk2LSRz+0AfLd1uVj Wj4isowUSgruR21ZyR1i9hcjiP/r4DgI9mHGiXmg1z+6APRnVVyOVWH9SYP+bO99dyDU e9Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:to:references:from:subject:message-id:date :user-agent:mime-version:in-reply-to; bh=Wr3czjD8C+TqABEdjmtS88VIcs0aGhY8SkwUYq+/tr0=; b=357IbOk1irKqfFXbBCcaC+P1esSUvRMC12i/mQS/dkAhKbX/+VAqPvZ2Jfy5Sv/yOk k31xI5XuTsz2OVBJWSclI5H3580VL0dlpdpge5X9V4v1Ji0I/vAXv6Veb1qEKFb9ggoc xwbP1G4xO11Ayz8sB5PpGeT3jV3pwF5WQAL9bZOAxmS21gJJWKtAv/DcLyW8MONzaQ3J 2VeHx8krLb6ETE3sFciO/0JP8yEwyyKJY06NkGj9zZqxB8abCqNJbAsN4qMyNNdCjacG u9IbU+lPDJzmMof/9znVr9u7EiXpiFmNeCMB83M2/TJnGmb/g36QqXnnW6822BZY3ejb avDQ== X-Gm-Message-State: AOAM530ea5GoiS450sGWGlnc+sZSDV/X30DepDqIgM98lWgmoehFT9aV xj+32szXlx37xKIAtgdKk40di6QCYnA/nw== X-Google-Smtp-Source: ABdhPJwzMKprqSwGPG/gPc54ZZCo81lfV2l4/yM8AvcVluqZq0xshgmw0SoFAz136BRT9XtspK74dg== X-Received: by 2002:a17:906:b88f:: with SMTP id hb15mr14578504ejb.91.1636030826404; Thu, 04 Nov 2021 06:00:26 -0700 (PDT) Received: from [172.17.100.137] ([212.48.107.10]) by smtp.gmail.com with ESMTPSA id cw10sm2745091ejc.80.2021.11.04.06.00.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 04 Nov 2021 06:00:25 -0700 (PDT) To: pf@freebsd.org References: From: Kajetan Staszkiewicz Subject: Re: "pfctl: Cannot allocate memory" issue with a large table Message-ID: Date: Thu, 4 Nov 2021 14:00:17 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G" X-Rspamd-Queue-Id: 4HlNzJ4vZxz4mYY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuxpowered-net.20210112.gappssmtp.com header.s=20210112 header.b=bDagGbQV; dmarc=none; spf=pass (mx1.freebsd.org: domain of vegeta@tuxpowered.net designates 2a00:1450:4864:20::535 as permitted sender) smtp.mailfrom=vegeta@tuxpowered.net X-Spamd-Result: default: False [-3.23 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tuxpowered-net.20210112.gappssmtp.com:s=20210112]; NEURAL_HAM_MEDIUM(-0.63)[-0.630]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[pf@freebsd.org]; TO_DN_NONE(0.00)[]; HAS_ATTACHMENT(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[tuxpowered.net]; NEURAL_SPAM_SHORT(1.00)[1.000]; DKIM_TRACE(0.00)[tuxpowered-net.20210112.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::535:from]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G Content-Type: multipart/mixed; boundary="TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT"; protected-headers="v1" From: Kajetan Staszkiewicz To: pf@freebsd.org Message-ID: Subject: Re: "pfctl: Cannot allocate memory" issue with a large table References: In-Reply-To: --TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 23.10.21 20:42, Marcel Bischoff wrote: > I often run into the "pfctl: Cannot allocate memory" error upon > replacing the table contents. Hi, I've encountered a similar issue after upgrading to FreeBSD 13.0. I have even cherry-picked https://github.com/freebsd/freebsd-src/commit/ea21980a3facfed4c2c6fd10d0f= 16276564fb540 which has not helped. I have a theory what is the problem here, but I lack detailed knowledge to confirm it. I have multiple Load Balancers running FreeBSD 11 or 13 and some of them run with with only 6 or 8GiB of memory installed. Each one shows 1-3GiB "wired" memory, <200MiB "active" memory and "inactive" slowly occupying all available memory within weeks after boot. Once there is only a few hundred MiB free memory, I can't reload the pf ruleset anymore on FreeBSD 13. Most of memory allocations in pf happens with M_NOWAIT flag. The aforementioned patch changes IOCTLs to request memory with M_WAITOK, but this does not change memory allocated for tables themselves. My guess is that when memory is full of inactive pages, it becomes impossible to allocate more UMA objects with M_NOWAIT, as it would require first getting rid of those pages (swapping them out? freeing them?). I'm unsure if this is due to changes in pf between 11 and 13, or rather increased memory pressure from other parts of system. I've always thought that it is beneficial to keep as much buffers / caches / inactive stuff in memory for better performance, but apparently it makes allocations which can't wait fail. Or at least that's my best guess, which somebody more experienced in in-kernel memory management (as I understand this would never be an issue in userspace!) should verify. --=20 | pozdrawiam / greetings | Powered by macOS, Debian and FreeBSD | | Kajetan Staszkiewicz | www: http://vegeta.tuxpowered.net | `------------------------^--------------------------------------' --TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT-- --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wmMEABEIACMWIQSOEQZObv2B8mf0JbnjtFCvbXs6FAUCYYPZYQUDAAAAAAAKCRDjtFCvbXs6FAJ6 AKDuvPAFsr+wtI4tUWwK0YZnmRWFIwCcCJ/5ta5gAiROyX5uRVf+CDkQqRY= =AXFW -----END PGP SIGNATURE----- --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G--