From nobody Thu Dec 23 14:39:03 2021
X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B2F3618FC420;
	Thu, 23 Dec 2021 14:39:06 +0000 (UTC)
	(envelope-from kp@FreeBSD.org)
Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "smtp.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4JKXrV2Nspz4qsY;
	Thu, 23 Dec 2021 14:39:06 +0000 (UTC)
	(envelope-from kp@FreeBSD.org)
Received: from venus.codepro.be (venus.codepro.be [5.9.86.228])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mx1.codepro.be", Issuer "R3" (verified OK))
	(Authenticated sender: kp)
	by smtp.freebsd.org (Postfix) with ESMTPSA id 18074CD39;
	Thu, 23 Dec 2021 14:39:06 +0000 (UTC)
	(envelope-from kp@FreeBSD.org)
Received: by venus.codepro.be (Postfix, authenticated sender kp)
 id 6610243B3D;
	Thu, 23 Dec 2021 15:39:04 +0100 (CET)
From: Kristof Provost <kp@FreeBSD.org>
To: freebsd-arm <freebsd-arm@freebsd.org>
Cc: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: pf cannot allocate memory after a time
Date: Thu, 23 Dec 2021 15:39:03 +0100
X-Mailer: MailMate (1.14r5852)
Message-ID: <1CEFA3DA-2D46-45B3-ADE6-881A94A4D2E2@FreeBSD.org>
In-Reply-To: <YbTOficBUC8vhklu@ceres.zyxst.net>
References: <YbTOficBUC8vhklu@ceres.zyxst.net>
List-Id: Porting FreeBSD to ARM processors <freebsd-arm.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-arm
List-Help: <mailto:freebsd-arm+help@freebsd.org>
List-Post: <mailto:freebsd-arm@freebsd.org>
List-Subscribe: <mailto:freebsd-arm+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-arm+unsubscribe@freebsd.org>
Sender: owner-freebsd-arm@freebsd.org
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_="
Content-Transfer-Encoding: 8bit
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org;
	s=dkim; t=1640270346;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=rcGL2FhY3AT4HxuXziDCXzcBVhWzlgT0Lmq/cgo4RzE=;
	b=s16NLDqvWo8BTDsooyxtW25c70mGuxXlVHjP+2YvlP8YI+vId94Lsa5V9QLlijtWlq8eMT
	GFpisSOG1kSa2s4HMMsMxjfjRUsN6c38Ovl1XqowiW/q3a7Bdb8PAk9nTth4NVd0Nz7WOv
	Om47LAlM2MOE6SSbAVnEc03jUpM067IrFyzAdj+8p8KCScLqoIE9gmZJywdYqEWi6IgNCu
	Ak8GoAdldNLdG7fT67ka0PDJyzVrcWvDounefxtt/BMh9V6/kDvNtgsqEKZSzDkZsN+PEv
	zh7ofz7srtm5A8dyFqdIf8x1yGkqmOgyY4N6+/VcIyfnxxBDRXuDHgNfqi1vTw==
ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1640270346; a=rsa-sha256; cv=none;
	b=kNhj1SrIw4hngntKe4te345sCvz+l/A3bu5JYoVi0rNJLvs4+yrEXEsLwbGZ0K042uROdV
	vygMVPwRzB3JngflAMw/wlJKqUcilR2TLJZ/EWMmsILL2QlcEvxTbmz2l1DZg5Sn1Roqsu
	WofbOYkCcKCwQcRy86+3KweQ3NBcrm7rqmaEKFMlfubeLIX8cUdzb7QowME0JsVwOGUboX
	hNORHy5pZRfYoQponh6JmDkduLSUOeAynspiuCutAER+heVMY11ZfBxr0hsU2Wr5EsfJlF
	jBsISKjS/cKBza4FNu9ciRwZPWsNRhGE05k7+Q+ZAfv8f2rikZ6/Pn4xNxoB7g==
ARC-Authentication-Results: i=1;
	mx1.freebsd.org;
	none
X-ThisMailContainsUnwantedMimeParts: N


--=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_=
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 11 Dec 2021, at 17:14, tech-lists wrote:
> context: main-n251261-25d0ccbe101 on arm64.aarch64 (raspberry 
> pi4b/8GB)
…
> pfctl: Cannot allocate memory.
>
<long story snipped>

See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406

Tl;dr: pfctl fails to add addresses to a table because an uma_zalloc() 
call fails).

vmstat -z output:

	ITEM                   SIZE  LIMIT     USED     FREE      REQ     
FAILSLEEP XDOMAIN
	pf table entries:       160, 400000,   15196,    1604,   36039,   9,   
0,   0

We can clearly see the memory allocation failure there, but no obvious 
reason why.
It appears that there’s plenty of free memory, and we’re also 
clearly far away from the configured limit. Nevertheless the 
uma_zalloc() call returns NULL.

I’ve tried to dtrace that, but ran into impossible results from dtrace 
(the uma_zalloc_args:return probe did not fire, but the 
pfr_create_kentry:return did. pfr_create_kentry() unconditionally calls 
uma_zalloc_args(), so that shouldn’t be possible).

Right now the suspicion is that there’s something strange going on 
with the arm64 allocator (because it’s not been seen on amd64 so far), 
but I’m generally uncertain of everything, other than it’s not 
actually pf’s fault that it’s not getting memory.

Best regards,
Kristof
--=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_=
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"=
>
</head>
<body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty=
le=3D"white-space: normal;">
<p dir=3D"auto">On 11 Dec 2021, at 17:14, tech-lists wrote:</p>
</div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote=
 style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136=
BCE; color: #136BCE;"><p dir=3D"auto">context: main-n251261-25d0ccbe101 o=
n arm64.aarch64 (raspberry pi4b/8GB)</p>
</blockquote></div>
<div class=3D"markdown" style=3D"white-space: normal;">
<p dir=3D"auto">=E2=80=A6</p>
</div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote=
 style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136=
BCE; color: #136BCE;"><p dir=3D"auto">pfctl: Cannot allocate memory.</p>
<br></blockquote></div>
<div class=3D"markdown" style=3D"white-space: normal;">
<p dir=3D"auto">&lt;long story snipped&gt;</p>
<p dir=3D"auto">See also <a href=3D"https://bugs.freebsd.org/bugzilla/sho=
w_bug.cgi?id=3D260406">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D=
260406</a></p>
<p dir=3D"auto">Tl;dr: pfctl fails to add addresses to a table because an=
 uma_zalloc() call fails).</p>
<p dir=3D"auto">vmstat -z output:</p>
<pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; border=
: thin solid gray; overflow-x: auto; max-width: 90vw; background-color: #=
E4E4E4;"><code>ITEM                   SIZE  LIMIT     USED     FREE      =
REQ     FAILSLEEP XDOMAIN
pf table entries:       160, 400000,   15196,    1604,   36039,   9,   0,=
   0
</code></pre>
<p dir=3D"auto">We can clearly see the memory allocation failure there, b=
ut no obvious reason why.<br>
It appears that there=E2=80=99s plenty of free memory, and we=E2=80=99re =
also clearly far away from the configured limit. Nevertheless the uma_zal=
loc() call returns NULL.</p>
<p dir=3D"auto">I=E2=80=99ve tried to dtrace that, but ran into impossibl=
e results from dtrace (the uma_zalloc_args:return probe did not fire, but=
 the pfr_create_kentry:return did. pfr_create_kentry() unconditionally ca=
lls uma_zalloc_args(), so that shouldn=E2=80=99t be possible).</p>
<p dir=3D"auto">Right now the suspicion is that there=E2=80=99s something=
 strange going on with the arm64 allocator (because it=E2=80=99s not been=
 seen on amd64 so far), but I=E2=80=99m generally uncertain of everything=
, other than it=E2=80=99s not actually pf=E2=80=99s fault that it=E2=80=99=
s not getting memory.</p>
<p dir=3D"auto">Best regards,<br>
Kristof</p>

</div></div></body>

</html>

--=_MailMate_44FD825A-A4A3-4FF5-A5A9-8E209A467CEF_=--