From nobody Mon Oct 31 05:47:28 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N12HN1GD5z4glqG for ; Mon, 31 Oct 2022 05:47:44 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from mail-yb1-xb32.google.com (mail-yb1-xb32.google.com [IPv6:2607:f8b0:4864:20::b32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4N12HM1Wg6z3ph5 for ; Mon, 31 Oct 2022 05:47:43 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: by mail-yb1-xb32.google.com with SMTP id f205so12586923yba.2 for ; Sun, 30 Oct 2022 22:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=aZyTYlN1WeKBSn5D6/lHUkwHpUvyeo0KhOWMYPjBwAE=; b=H8J4kNP7g+p9UbXH4qjSZRsASx/FBVhp1fPrJHsJMTVAPZe+I7GrY4cspQ2+GmG9Vr ph2XhbrKfh36Fv0UYZYQm5xB8GqjhDej3fZ10D1E7/K/ybmJOx625S8UT0+fMFikp1jV y3/QCB6Nof+2la42v3hE4qV0QlFKYQ5FpuFyNKzF7NaOd0XsxylZwf1uIJZFM46GIQaD ZfHG8MnRE6jR/YoSzazpSVEEZ2yfgowvy9HKF+210m+jWJgsw/EhOGPvJRyQHoDVvthF e1l8uQnNAb8Rrho1+AzOzZLrMFjGzfx4sGHmase+tHkbRjIAhUXqpTjIMVn/8QhKgcjj siJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aZyTYlN1WeKBSn5D6/lHUkwHpUvyeo0KhOWMYPjBwAE=; b=jSD+T2SLqugmZCAyaMY5chiM9mimitaHRR9klRz7sPslQ3jb0dd9glJQJBFfV1TERA GonNLt6bZqJY0djaqHQje5OSi092SX2N9DrgvSNscWq2w9vri41pBjsZ57dpDI3PSOsH p7uHcQqE+AfwQijJ9Gcrr32DTmf8FgWwLBsJOS6eHbahRBQXBJRU5XCaGByjJo1VTdAi amLj2PXNGOeYtv70ErmhRAYjOHTIzj9x7THXCYwdoBx0dgz38YHsDg28Jj7+0nlr9nNz cHgZaIZk1LIMJQE0UNgSFzL0ALHSrqlggUo9/Z/YSnOyXuIELYVCJqS9VGM1FfecZiWD TzWQ== X-Gm-Message-State: ACrzQf3EH5Qq+n/qD8XLtSBoABF//fdDkh8opq5t0/TaA7fh4GQJ7ZDZ KGFnv3hkj0gmc/wLw0XUnEpvVJV/8FyVS4iXDM/nqNw1 X-Google-Smtp-Source: AMsMyM4te2aorhd3CYpsPhdouu/Aw8V/6n0k1xgSonX+u2w/NSwwjIA1b7KxGy0aDo8mKOVQaNkwqE5aza+prDowSs8= X-Received: by 2002:a5b:311:0:b0:6c3:b37b:a165 with SMTP id j17-20020a5b0311000000b006c3b37ba165mr10759305ybp.467.1667195262222; Sun, 30 Oct 2022 22:47:42 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> In-Reply-To: From: Archimedes Gaviola Date: Mon, 31 Oct 2022 13:47:28 +0800 Message-ID: Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build To: Mark Millard Cc: freebsd-current Content-Type: multipart/alternative; boundary="00000000000060449305ec4e25ea" X-Rspamd-Queue-Id: 4N12HM1Wg6z3ph5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=H8J4kNP7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of archimedes.gaviola@gmail.com designates 2607:f8b0:4864:20::b32 as permitted sender) smtp.mailfrom=archimedes.gaviola@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b32:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --00000000000060449305ec4e25ea Content-Type: text/plain; charset="UTF-8" > Okay noted on GPT not MBR method with gpart. > I did not happen to have a MBR example around. So I could > only show GPT. The note was more to avoid confusion than > anything, since the two are not equivalent for how they > work. > Okay, this is noted. > > > By the way, what's the proper allocation size of swap in FreeBSD? > > FreeBSD has a waring that it produces indicating possible mistuning > when you potentially have too much. An example is: > > warning: total configured swap (2097152 pages) exceeds maximum recommended > amount (916632 pages). > warning: increase kern.maxswzone or reduce amount of swap. > > The numbers are dependent on the amount of RAM present and > other details. > > My understanding is that increasing kern.maxswzone has tradeoffs. > I avoid getting the message because I do not understand the > tradeoffs or how to manage the tradeoffs or even how to identify > an instance of hitting such a tradeoff. > Basically the warning messages you've shared are the messages I encountered with my older FreeBSD system running on MIPS32 at the time I allocated a swap partition because of the higher allocation size I've made. So what I did is gradually adjust the swap size until such warnings disappear. I did not go through the details as most likely it requires a deeper knowledge on this area. That's why this experience illuminated me again with my RPi 3B ARM system on the proper allocation size. But yes, below you have the allocation size. > > For aarch64 I've been about to have swap of about 3.4 to 3.5 or > so times the amount of RAM without getting the warnings. That > is why 3.5G in my RPi3B example. (So RAM+SWAP approx.= 4.5*RAM.) > (armv7 only allows more like 1.8 times the RAM before getting > the warning.) > Okay this is noted. I'll take the 3.5G size as this is based on your actual experience. > > I avoid even getting too close to the warning as there seems to > be some build-to-build variability in what fits vs. not. This > avoids having to frequently adjust the size. > > I, too, need to avoid such warnings as much as possible with this RPi 3B configuration. > Going from the other side, how much RAM+SWAP will your activities > use? To avoid accurately figuring out such, you may just want to > have near the 3.4 to 3.5 times RAM. (There have been times when > clang had memory use oddities that required more than normal for > a time, for example.) > I'll just follow the size you have and let me observe how it goes. > > > This RPi 3B has 1GB of RAM (~947 MB), do I need to set twice the > capacity of this physical RAM? > > Ultimately your choice. How much parallel activity you > want to attempt likely contributes. If you build ports, > you might do so in a way that uses more RAM+SWAP than > system builds do, for example. > Okay this is noted. For now, building the kernel and world is my goal, no ports yet. > > > (Note: swap file usage is subject to deadlock conditions > > avoided by use of swap partitions.) > > > > This is noted. > > > > > > I use a serial console & ssh session only context to avoid > > having sizable competition for RAM. > > > > I avoid using tmpfs because it competes for RAM use. > > > > I use the likes of ( in, say, /boot/loader/conf ): > > > > # > > # Delay when persistent low free RAM leads to > > # Out Of Memory killing of processes: > > vm.pageout_oom_seq=120 > > > > This delays potential "killed: failed to reclaim memory" kills, > > possibly long enough to reach a state where sufficient memory is > > reclaimed. > > > > Alright this is well noted too. > > There is tuning related to "a thread waited too long to > allocate a page" that happens because of paging I/O > characteristics. But but I've not hit that type of > error. > > I'll also note that the "out of swap space" case is a > misnomer in that it is one or two of 2 internal data > structures that is out of space, not necessarily the > swap space on the media. Again, I've not ever hit that > type of error. I'm not aware of tuning for this case. > Okay, noted as well on this info. Let me just try the 3.5G swap allocation. I will post another thread if I ever encounter these types of errors. > > > I'll note that the status "killed: failed to reclaim memory" does > > not require that swap be used much at all. Sustained low free RAM > > from just one process that always stays runnable and has a > > sufficiently large active set of pages can be sufficient to end up > > with such kills. Having swap allows for inactive pages to get out > > of the way, which can help. > > > > I use the likes of ( in, say, /etc/ssyctl.conf ): > > > > # > > # Together this pair avoids swapping out the process kernel stacks. > > # This avoids processes for interacting with the system from being > > # hung-up. > > vm.swap_enabled=0 > > vm.swap_idle_enabled=0 > > > > This allows paging to the swap space but disallows moving > > kernel thread stacks to the swap space. Otherwise the > > processes used to interact with the RPi3 can become > > non-runnable, preventing such interactions. > > > > Okay this too is well noted. > > > > > > I have NVMe or SSD based USB media, not microsd cards nor > > spinning rust. (I use just bootcode.bin and timeout files > > on microsd media for the RPi3B. Even the rest of the RPi* > > firmware is on the USB media, as well as u-boot.bin .) > > This may contribute to why I've never gotten a "a thread > waited too long to allocate a page" on any system. (Some > systems, while bootable via USB3 media I have, also have > have even faster internal media that is normally used.) > Alright so there's significance. > > > My usage of such a configuration struture for building > > software (world, kernel, ports) applies to all the > > systems I do such with, including ones with a lot more > > resources, including a lot more RAM. > > > > Thanks for these inputs, noted on these things! I haven't tried NVMe and > SSD media in my RPi 3B. So, they are far more superior as compared to > microSD cards when it comes to building software? > > My understanding is that microsd card media is fairly > generally not as good for such contexts: slower, fails > sooner, etc. > I'll take note of this one as I may encounter those attributes along the course of building software. It's something that I need to explore and do some research ahead. > > I happen to boot multiple types of machines from the > same media so I use USB3 media that is compatible with > USB2 use, a single such USB3 device not needing a > powered hub for use on the likes of an RPi3B. (Lots > of USB3 media around would require external power for > USB2 or an RPi3B use.) I need a powered hub for 2 or > more such media on a RPi3B. > Okay, that's right. In my experience, inserting some devices tends to reset the 4 USB ports' power, thus to prevent such behavior needs a self-powered hub. Thanks and best regards, Archimedes --00000000000060449305ec4e25ea Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

> Okay noted on GPT not= MBR method with gpart.


I did not happen to have a MBR example around. So I could
only show GPT. The note was more to avoid confusion than
anything, since the two are not equivalent for how they
work.

Okay, this is noted.
= =C2=A0

> By the way, what's the proper allocation size of swap in FreeBSD?<= br>
FreeBSD has a waring that it produces indicating possible mistuning
when you potentially have too much. An example is:

warning: total configured swap (2097152 pages) exceeds maximum recommended = amount (916632 pages).
warning: increase kern.maxswzone or reduce amount of swap.

The numbers are dependent on the amount of RAM present and
other details.

My understanding is that increasing kern.maxswzone has tradeoffs.
I avoid getting the message because I do not understand the
tradeoffs or how to manage the tradeoffs or even how to identify
an instance of hitting such a tradeoff.

Basically the warning messages you've shared are the messages I encoun= tered with my older FreeBSD system running on MIPS32 at the time I allocate= d a swap partition because of the higher allocation size I've made. So = what I did is gradually adjust the swap size until such warnings disappear.= I did not go through the details as most likely it requires a deeper knowl= edge on this area. That's why this experience illuminated me again with= my RPi 3B ARM system on the proper allocation size. But yes, below you hav= e the allocation size.
=C2=A0

For aarch64 I've been about to have swap of about 3.4 to 3.5 or
so times the amount of RAM without getting the warnings. That
is why 3.5G in my RPi3B example. (So RAM+SWAP approx.=3D 4.5*RAM.)
(armv7 only allows more like 1.8 times the RAM before getting
the warning.)

Okay this is noted. I'= ;ll take the 3.5G size as this is based on your actual experience.
=C2=A0

I avoid even getting too close to the warning as there seems to
be some build-to-build variability in what fits vs. not. This
avoids having to frequently adjust the size.


I, too, need to avoid such warnings as= much as possible with this RPi 3B configuration.
=C2=A0
Going from the other side, how much RAM+SWAP will your activities
use? To avoid accurately figuring out such, you may just want to
have near the 3.4 to 3.5 times RAM. (There have been times when
clang had memory use oddities that required more than normal for
a time, for example.)

I'll just fol= low the size you have and let me observe how it goes.
=C2=A0
<= /div>

> This RPi 3B has 1GB of RAM (~947 MB), do I need to set twice the capac= ity of this physical RAM?

Ultimately your choice. How much parallel activity you
want to attempt likely contributes. If you build ports,
you might do so in a way that uses more RAM+SWAP than
system builds do, for example.

Okay thi= s is noted. For now, building the kernel and world is my goal, no ports yet= .
=C2=A0=C2=A0

> (Note: swap file usage is subject to deadlock conditions
> avoided by use of swap partitions.)
>
> This is noted.
>=C2=A0
>
> I use a serial console & ssh session only context to avoid
> having sizable competition for RAM.
>
> I avoid using tmpfs because it competes for RAM use.
>
> I use the likes of ( in, say, /boot/loader/conf ):
>
> #
> # Delay when persistent low free RAM leads to
> # Out Of Memory killing of processes:
> vm.pageout_oom_seq=3D120
>
> This delays potential "killed: failed to reclaim memory" kil= ls,
> possibly long enough to reach a state where sufficient memory is
> reclaimed.
>
> Alright this is well noted too.

There is tuning related to "a thread waited too long to
allocate a page" that happens because of paging I/O
characteristics. But but I've not hit that type of
error.

I'll also note that the "out of swap space" case is a
misnomer in that it is one or two of 2 internal data
structures that is out of space, not necessarily the
swap space on the media. Again, I've not ever hit that
type of error. I'm not aware of tuning for this case.
<= div>
Okay, noted as well on this info. Let me just try the 3.= 5G swap allocation. I will post another thread if I ever encounter these ty= pes of errors.
=C2=A0

> I'll note that the status "killed: failed to reclaim memory&q= uot; does
> not require that swap be used much at all. Sustained low free RAM
> from just one process that always stays runnable and has a
> sufficiently large active set of pages can be sufficient to end up
> with such kills. Having swap allows for inactive pages to get out
> of the way, which can help.
>
> I use the likes of ( in, say, /etc/ssyctl.conf ):
>
> #
> # Together this pair avoids swapping out the process kernel stacks. > # This avoids processes for interacting with the system from being
> # hung-up.
> vm.swap_enabled=3D0
> vm.swap_idle_enabled=3D0
>
> This allows paging to the swap space but disallows moving
> kernel thread stacks to the swap space. Otherwise the
> processes used to interact with the RPi3 can become
> non-runnable, preventing such interactions.
>
> Okay this too is well noted.
>=C2=A0
>
> I have NVMe or SSD based USB media, not microsd cards nor
> spinning rust. (I use just bootcode.bin and timeout files
> on microsd media for the RPi3B. Even the rest of the RPi*
> firmware is on the USB media, as well as u-boot.bin .)

This may contribute to why I've never gotten a "a thread
waited too long to allocate a page" on any system. (Some
systems, while bootable via USB3 media I have, also have
have even faster internal media that is normally used.)

Alright so there's significance.
=C2=A0
<= /div>

> My usage of such a configuration struture for building
> software (world, kernel, ports) applies to all the
> systems I do such with, including ones with a lot more
> resources, including a lot more RAM.
>
> Thanks for these inputs, noted on these things! I haven't tried NV= Me and SSD media in my RPi 3B. So, they are far more superior as compared t= o microSD cards when it comes to building software?

My understanding is that microsd card media is fairly
generally not as good for such contexts: slower, fails
sooner, etc.

I'll take note of this= one as I may encounter those attributes along the course of building softw= are. It's something that I need to explore and do some research ahead.<= br>
=C2=A0

I happen to boot multiple types of machines from the
same media so I use USB3 media that is compatible with
USB2 use, a single such USB3 device not needing a
powered hub for use on the likes of an RPi3B. (Lots
of USB3 media around would require external power for
USB2 or an RPi3B use.) I need a powered hub for 2 or
more such media on a RPi3B.

Okay, that's= right.=C2=A0 In my experience, inserting some devices tends to reset the 4= USB ports' power, thus to prevent such behavior needs a self-powered h= ub.

Th= anks and best regards,
Archimedes
= --00000000000060449305ec4e25ea--