From nobody Mon Nov 21 03:48:58 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NFth11WT9z4hCkq for ; Mon, 21 Nov 2022 03:50:09 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NFth00wxhz46WM for ; Mon, 21 Nov 2022 03:50:08 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=BxsDfZgj; spf=pass (mx1.freebsd.org: domain of archimedes.gaviola@gmail.com designates 2607:f8b0:4864:20::1136 as permitted sender) smtp.mailfrom=archimedes.gaviola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-3a4c2c83300so6312867b3.13 for ; Sun, 20 Nov 2022 19:50:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LyEgcMDewvLIc1qY+Siti4IMendDb8bUL2v1yYiVc9k=; b=BxsDfZgjJYr9SvEBClsadrmFUc6yK8VEdLOJPyZhjy0fq2RJmlWKjXMvvlShsjdYws jQhRQffHNqV9+AEVPzv8LDYG4RhL59s4OGLaGPuIrPgis3O8r6mnOt5NDQqiYTVNlzZn TJj9pn+QhRHrv6hOZTbscEESpZaOmAPfCG/hiKRA/x0GErVbMp2GzQwOBQsIqfNjy7Dg MOu7uoTkhjXbxqS+lQLVdPkEUKr7DG3ZCDkZY+1ahgbRt6omaClNIdtcwFOlY7HhoKp2 Bz0YCaErah3oTa613ozGbBkDckJZ18mFjGJ16LH36bOP0PN+MGIOTA/Mt/T642B7W68s pduQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LyEgcMDewvLIc1qY+Siti4IMendDb8bUL2v1yYiVc9k=; b=pTL9S7MRxFa/6XfAUln93tgd0mK1xYo4/2jkHxK8f2mXOUT+MwHAHDPdd2+Rjlv4x5 zEX7+vLxmIT3KWwdgmiVKx8bNnEs4DY2xXQZAMbPMTZ85D1zTqEJpHR4HnxCQanOdyuc oCgbUwYAu/3GLeIOJ3vSalGfpi106B7JxRmRK1nh4Q5g4a4sfsUSq0imc4kFaXjFX7+p lENZ+jLodf8tk9fwoucJ4DF2ySvne+dlac/6SZ5Dfxev0Rg9z/fT12ofvQeREwVqPCtU YlVIlmENwngOeewWUP8TkX+ghGxzthQxrtNJIJecXBAmfw24IPP8E7/ECJ36VfOARJ/W YJIQ== X-Gm-Message-State: ANoB5pmfmLYH9HypjoYc6BsLGpxZaMFuvy44Z4RAIMFG78fVrw6Kck9m nzqB/hL0c6SEjT0RFZxX/2Td39ao3IVEwdlOO8jY5XTRu3o= X-Google-Smtp-Source: AA0mqf6g0UtGXx/+87KUfp+6aWEH4PmIH5Z0hNAOEOnq4uu4n8QOSU5xRGv0rmBkbKFvMIGYqscoaEIcywI4co+/DI4= X-Received: by 2002:a81:ff09:0:b0:367:6bb7:9ba9 with SMTP id k9-20020a81ff09000000b003676bb79ba9mr15181689ywn.231.1669002606565; Sun, 20 Nov 2022 19:50:06 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <1722758786.127406.1667909706281@localhost> <1E33E6A3-ABB8-4804-B2A2-0E95E853C860@yahoo.com> In-Reply-To: From: Archimedes Gaviola Date: Mon, 21 Nov 2022 11:48:58 +0800 Message-ID: Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build To: Mark Millard Cc: Ronald Klop , freebsd-current Content-Type: multipart/alternative; boundary="0000000000007e5df705edf2f368" X-Spamd-Result: default: False [-2.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.987]; NEURAL_SPAM_MEDIUM(0.54)[0.540]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1136:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4NFth00wxhz46WM X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --0000000000007e5df705edf2f368 Content-Type: text/plain; charset="UTF-8" On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola < archimedes.gaviola@gmail.com> wrote: > > > On Wed, Nov 9, 2022 at 1:37 AM Mark Millard wrote: > >> On Nov 8, 2022, at 04:15, Ronald Klop wrote: >> >> > Van: Warner Losh >> > Datum: dinsdag, 8 november 2022 04:28 >> > Aan: Archimedes Gaviola > > . . . >> > ... >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: >> 40960 >> > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to >> allocate a page >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: >> 28672 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 8192 >> > This means that paging to the swap partition and/or swap file took >> too long (> 30 seconds... that's all that indefinite means). It also means >> that it can't write to backing store dirty pages to give to another >> process... >> > Typical reason is that the disk / flash is not responsive to writes >> for some reason. You'll need to find why... I'd look at trims. >> > Or.... if you can't change the disk... you need to put less memory >> pressure on it.. >> > Warner >> > >> > >> > >> > NB: a way to put less memory pressure on it is not using -j3, but -j2 >> or -j1 in your make command. >> > >> > > Hi Mark, > > >> Extending Ronold's comment: If things are really taking this >> long for the paging I/O, you might actually find, say, -j2 >> takes less elapsed time than -j3 because of the latencies >> involved in -j3 causing more overall delay. >> > > Yes I'll take these options on lowering down N in the -jN parameter as my > next steps. So far so good with -j3, ongoing build is still observed for 17 > hours now. > > >> >> vm.pfault_oom_attempts=-1 would still be appropriate for avoiding >> I/O kills at any -jN: the smaller -jN just makes the issue less >> likely, not impossible. (Again, presuming sufficient swap/paging >> space if deadlock is to be well avoided.) >> > > The ongoing build is at the moment on > /usr/src/contrib/llvm-project/llvm/lib/*. I'm observing from time-to-time > if the error will occur again. > > >> (I use NVMe or SSD USB media that do not get such long delays but >> fit the power limitations of the context. I have about as little >> on microsd card media as I can get away with in my context. I also >> avoid spinning rust. Thus I've only gotten "indefinite wait buffer" >> or the like back before such was true, long ago.) >> > > Okay thanks for sharing this one. Keeping this in my mind just in case I > needed these types of media soon. > > Thanks and best regards, > Archimedes > Hi Mark, As a recap on the kernel tunables, the changes are the following, root@generic:~ # sysctl -a | grep oom vm.pageout_oom_seq: 120 vm.pfault_oom_wait: 10 vm.pfault_oom_attempts: -1 With -j1 and -j2 options, both were able to complete the kernel and buildworld compilation in 103 and 84 hours respectively. Though I still could see messages on "swap_pager: indefinite wait buffer: bufobj" but definitely it's ignorable as it survived the compilation process. With the -j3 option, it failed along the course of compilation, it encountered the previous error on "failed to reclaim memory" but this time this error is not that relevant as -j1 and -j2 already works. Preferably with -j2 as the appropriate choice for my RPi 3B build setup. Thanks and best regards, Archimedes --0000000000007e5df705edf2f368 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Nov 9, 2022 at 10:15 AM Archi= medes Gaviola <archimede= s.gaviola@gmail.com> wrote:


On Wed, Nov 9, 202= 2 at 1:37 AM Mark Millard <marklmi@yahoo.com> wrote:
On Nov 8, 2022, at 04:15, Ronald Klop <ronald-lists@klop.ws<= /a>> wrote:

> Van: Warner Losh <imp@bsdimp.com>
> Datum: dinsdag, 8 november 2022 04:28
> Aan: Archimedes Gaviola <archimedes.gaviola@gmail.com
> . . .
> ...
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 40= 96
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096=
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: 40= 960
> pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to= allocate a page
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28= 672
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192=
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 40= 96
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 81= 92
>=C2=A0 =C2=A0This means that paging to the swap partition and/or swap f= ile took too long (> 30 seconds... that's all that indefinite means)= . It also means that it can't write to backing store dirty pages to giv= e to another process...
>=C2=A0 =C2=A0Typical reason is that the disk / flash is not responsive = to writes for some reason. You'll need to find why... I'd look at t= rims.
>=C2=A0 =C2=A0Or.... if you can't change the disk... you need to put= less memory pressure on it..
>=C2=A0 =C2=A0Warner
>=C2=A0 =C2=A0
>
>
> NB: a way to put less memory pressure on it is not using -j3, but -j2 = or -j1 in your make command.
>

Hi Mark,
=C2=A0
Extending Ronold's comment: If things are really taking this
long for the paging I/O, you might actually find, say, -j2
takes less elapsed time than -j3 because of the latencies
involved in -j3 causing more overall delay.

=
Yes I'll take these options on lowering down N in the -jN paramete= r as my next steps. So far so good with -j3, ongoing build is still observe= d for 17 hours now.
=C2=A0

vm.pfault_oom_attempts=3D-1 would still be appropriate for avoiding
I/O kills at any -jN: the smaller -jN just makes the issue less
likely, not impossible. (Again, presuming sufficient swap/paging
space if deadlock is to be well avoided.)

The ongoing build is at the moment on /usr/src/contrib/llvm-project/llvm= /lib/*. I'm observing from time-to-time if the error will occur again.<= br>
=C2=A0
(I use NVMe or SSD USB media that do not get such long delays but
fit the power limitations of the context. I have about as little
on microsd card media as I can get away with in my context. I also
avoid spinning rust. Thus I've only gotten "indefinite wait buffer= "
or the like back before such was true, long ago.)

=
Okay thanks for sharing this one. Keeping this in my mind just i= n case I needed these types of media soon.

Thanks = and best regards,
Archimedes

Hi Mark,

As a recap on the kerne= l tunables, the changes are the following,

roo= t@generic:~ # sysctl -a | grep oom
vm.pageout_oom_seq: 120
vm.pfault_= oom_wait: 10
vm.pfault_oom_attempts: -1

With -j= 1 and -j2 options, both were able to complete the kernel and buildworld com= pilation in 103 and 84 hours respectively. Though I still could see message= s on "swap_pager: indefinite wait buffer: bufobj" but definitely = it's ignorable as it survived the compilation process. With the -j3 opt= ion, it failed along the course of compilation, it encountered the previous= error on "failed to reclaim memory" but this time this error is = not that relevant as -j1 and -j2 already works. Preferably with -j2 as the = appropriate choice for my RPi 3B build setup.

Than= ks and best regards,
Archimedes
--0000000000007e5df705edf2f368--