From nobody Tue May 02 07:25:21 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9Wnv103bz48d87 for ; Tue, 2 May 2023 07:25:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9Wnt4Dzqz4XhT for ; Tue, 2 May 2023 07:25:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1683012335; bh=CN8wcZ/sNIHkcesWWuwQT00RWCKua7L8HAXtZbZzsW8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=qUgvxPrMSbCdbLRfufEdIxer/sIsCssQPKW4U7a7/pkgHyJJRpwx7odib6mxzT9vyv/nzMLG5bea8aZSJnyWdjck94jjnlQRwAn35tva8ImfWesS78NGLWPQSyr0YoprAmL9m/2GxKhTq3whPJPIOuvYbbcd/AMCKSdGHqYylhFYZn4cb2WG0zm4dKVKSrsx3u/ptOelb0r6jjI6HzNPF88ZIjtd/1co82pKrW8JJ/XOkoFa5H1ORLAsOUmStJd1w5KSOiH7Pjxw8lYUCgZBeWQftCl6ychV6+z8+L4+TqFaa+bCLGcKCkQz917ilaVlCpesOHyCNZH6bxiRdYVzdw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1683012335; bh=EQJQ8d92bPcKoUQVmVJAC48+CuooMk7pvXlTJUx9dGn=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=S4aPheFK7g2BPcgX86nCP2wckq1Lqy37HfB6aVjKMm79H2pyr4RNIxb8XtdE0akkchsX/JrGSjgRUj8vfA65XVp05S21seKqlB1m4gyEZndnXL4+SzybYzzSBTLI6pURjFj68HtxdCx6ahOQ2fFvButwA9IiZ8jGgNaew3dharERQOqhkqiIW7ojp4V5cCFYiFTaFFJMS8tiZAm3D7RcvzQHPR+BYCCqMspJC36y78lpGO0VHRzmjEDgdKBe63oU/LlzEnFWYrxHxavbzj/RK4ehNTHSE5k/U60CKyGR2OxaGuZe9hk+mRLhPue2yRd0cnF1Q2xeTARdiPOaaip+0g== X-YMail-OSG: tBQ4MaUVM1nWDyajyLqv.iAJslDv2BUZBYpkqxfaDaiRofW2vsNK.fqfux4DiHB xiuwxiqDPt.kBvMHXCBh3HMsE_LgPQsYJkwvtD1bpOESqTN1mHRzppCcUn16JBciQ8dgVRFrBPtB 81kJb9DPwW48ZqpzT4QGIDs6YpU.GuSuNpReo8DWc4gqmaXs34pCEVGAqStOcSmytjAirgSJqykl cG4sjXp_NV3STOK4zLWRKWYNNsJu_HRcQM71AvUlgDnWjIbaGPoJVAf.NSr00xRaBQ.2sRTYk_rz QtJiuNHwF8owMAuYmKDp.jGIDfAOnbYz5mbnMSFrQA_9D6fmA1DajzxtQcFZctpzg3wWOAy6FSix za.zT6yOYweMJRj1Qx4y0.ofGcJ0B_PRCx_c.kRmaXAdjgv4czS_ArXIo1s2ea1HfdGPmNrSuKxA ZtNrF2851ahuT0xY6xWLEkbeMRSemTk5U_CEPmGUe1y39FWd6LmShv8HrmWCRj5GzZJCmGbWUkpW ufy2W6N_Or_fIpo7AWL3nzWK2Mfl6A6nZjlP2P2pvUNd3VJZRORziyrpxnsjo0WkqTm6jLX9WfTj z0smLWPrbL51nqyTSSM75UWDXymNSwUZO.D6gapoQStMDEvhmP3sSPSmSxvu2FupxuUkNdp4Ttyn 30Iflea8Vogrjf2LQYfUA5BDLuxDjnN6Ef6dCwyALCkDeXYQgcorWP2kZeMJMPEJ44qlnNi6aMXT .6pSyYTFbhUYo8Y1kDeIHoVC3hbpoF_UMhMFbEKRwmM8gjo_iy7FtPpcSujoKdmO2gw9D6ozsyN8 SpHlk0dwcmnMUfXu1uNjibAZiDHwK8uHTNoHTYZhdEpayrnjQ0IFuJByeV9xZDICiAnlh39ExoML gk3ItNcJxyETbEJ6bUdTAYsorG7PPus7gwrsMPpA9Yl2m94yTU9UE3pOXtAGhycomVdZV56Msr.j bjCrVXGVZH2j6wE7J17neQdOXs3C6rRhs09u9sQu86h6MIwcZ9e0aVD7pBxpxtqnWkJ1dNloCuIu bTzdLLm1ZLavGb5rFy5v1QRXlAtF4QjkHNGSx7BfWYdQERusH_sfpkaQCb1oIGn4clGdMVXyam6W .raFoCZZNBix1jIIZz0caWn5QTWkmlGW93rGWKIza_9cHZ0NpMTqwafoqu9AQjDnq5ozUDYgMD4y E37TxunFwqRYVW_pJSqPEe5MjrenVJPf04sapltt.oQXIjzc2IQyjpcT8DzEdIHm1.8evZEc4jwA XAjBX184Sr.s3t0HQWfho0pr7MoYdW75oR5RCR1z_6b9jkTy5aD6JTszdpqLvBPOKyWBOIOmd87E cab8S72hhFzjb8cByJ1Wq.6VyCCflgODwCIrFJf5kAT34NTogQJ1fDdsqiLjFlEEShJ9TN6phbLc 3A0TvpPMTPSt4TC4ZkbbB3A7fp4x0dVwWblqp_RqEk7I834B2PaVz47.yXmeTVSb5_9.7zp1S5Ka e0vlbB5kXV_NsdYhUfpbOreT6t.D.4tS0_dyAZVJ3tKCJWYBvQyhgVl6VZl3zOutZlC.J3dy.KIv IYASL6_tXGMqEIm5OeQAQ7M4ymWNQgvHaudIQIBLXgsQ9V_TaIF6q4eSSeyf4r6w.ezxPzcUCar_ 5k9.zYTQnHMAONn1p_VHkyJvV59nfPl2isV7SeJv3o5f23HUXGnCh.hKfDZ6Gxbk7YMZmw7AcEJo Vw.uI7Qgn0B8hvl1oroRJSG5YIH8gpyAGD_.ju1YXfihqdkBwhOtQz1m_KUFMHpjRMw0RLrj0bF9 93fTQHIrt1LbZTgQF46fZBbKvqlAUt_vrem3FnrG5HF4BX.BEgFUfupKlcbYqFIGgU7uBPONU7Lq WtMH52f5hIzP.Jrpok6UNMKNLGwtah5cdxmQnxZRZdo2vIZDFNzFumqCvEMmnLIVQjsX4kKl9ZLZ Tbh834LOBqs2Ydiuu.ngCibvEJe_Y5eXGr97YwOBtzNDvJVpejp8p61VYOxfi8pZ0dU6I.2gh7oU HnaSh6HdxYnKWaQK1UcXt.2_aEWxPRsuNiCBeF126suingJJOg4Mm3aX7HxYBmuIXLjxMZhs7aZh LZT4pCEvy8RdbklwHS8fzIZyZSxNARTOnH7QCjGODtvsBcmA4VvRf0..VBhi4Jj4opo0YNGVsPz0 I35usEyddjbhvi.2iDwgOSXtlGw7nMLyzzGeBxnsNMQmrBYqDhrLIhiFnU0BUV84X9ZFySxsUVpw QnhBVRvN6Sk8Jadb3clgXTh0HhPQRzuELrx2NuGUpKpBESa79BAcX70G2WYiSOdTJc3OGD.JyKpo pG7g- X-Sonic-MF: X-Sonic-ID: e6a79966-e981-45d7-bff2-dc90294b2ab9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 2 May 2023 07:25:35 +0000 Received: by hermes--production-gq1-546798879c-l2qgj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d712bd5b766f49f099bceab17e826b06; Tue, 02 May 2023 07:25:32 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: pid xxxx (progname), jid 0, uid 0, was killed: failed to reclaim memory From: Mark Millard In-Reply-To: <13b3f66b-45ab-ac63-f84a-419d5b376fa8@c0decafe.de> Date: Tue, 2 May 2023 00:25:21 -0700 Cc: "freebsd-arm@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <13b3f66b-45ab-ac63-f84a-419d5b376fa8@c0decafe.de> To: Daniel X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4Q9Wnt4Dzqz4XhT X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On May 1, 2023, at 23:55, Daniel wrote: > I noticed that on my aarch64 boards I recently get a lot of 'pid xxxx = (progname), jid 0, uid 0, was killed: failed to reclaim memory' messages = in syslog. >=20 > This happens on a rockpro64 as well as a raspberry pi 3b, both running = 13.2. None of the boards is near its memory capacity (more like less = than 50% used). Are you counting RAM+SWAP as "memory capacity"? Just RAM? The message is strictly about maintaining a certain amount of free RAM. It turns out swap does not automatically avoid the issue for all contexts. QUOTE from back in 2022-Dec for another context with the problem: (I've not reworked the wording to your context but the points will probably be clear anyway.) This is the FreeBSD kernel complaining about the configuration not well matching the RPi3B+ workload. In essence, it was unable to achieve it targeted minimum amount of free RAM in the sort of time frame (really: effort) it is configured for. Depending on what you do, the FreeBSD defaults do not work well for 1 GiByte of RAM. Swap space alone is insufficient because FreeBSD does not swap out processes that stay runnable. Just one process that stays runnable using a working set that is as large as the fits RAM for overall operation will lead to such "failed to reclaim memory" kills. But, if you are getting this, you will almost certainly need a non-trivial swap space anyway. I have a starting point to recommend, configuring some settings. As I've no detailed clue for your context, I'll just provide the general description. A) I recommend a swap space something like shown in the below (from gpart show output): =3D> 40 1953525088 da0 GPT (932G) 40 532480 1 efi (260M) 532520 2008 - free - (1.0M) 534528 7340032 2 freebsd-swap (3.5G) . . . 67643392 1740636160 5 freebsd-ufs (830G) 1808279552 145245576 - free - (69G) This size (3.5 GiBytes or so) is somewhat below were FreeBSD starts to complain about potential mistuning from a large swap space, given the 1 GiByte of RAM. (I boot the same boot media on a variety of machines and have other swap partitions to match up with RAM sizes. But I omitted showing them.) It is easy to have things like buildworld or building ports end up with individual processes that are temporarily bigger than the 1 GiByte RAM. Getting multiple cores going can also lead to not fitting and needing to page. I'll note that I normally use USB3 NVMe media that also works with USB2 ports. My alternate is USB3 SSD media that works with USB2 ports. I avoid spinning rust and microsd cards. This limits what I can usefully comment on for some aspects of configuration related to the alternatives. B) /boot/loader.conf content: # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) If use of vm.pfault_oom_attempts=3D-1 is going to be inappropriate, I do not have background with figuring out a good combination of settings for vm.pfault_oom_attempts and vm.pfault_oom_wait . I'll note that vm.pageout_oom_seq is not a time --more like how many insufficient tries to reclaim RAM happen in sequence before an OOM kill is started (effort). 120 is 10 times the default. While nothing disables such criteria, larger figures can be used if needed. (I've never had to but others have.) C) /etc/sysctl.conf content: # # Together this pair avoids swapping out the process kernel stacks. # This avoids one way for processes for interacting with the system # from ending up being hung-up. vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 D) I strictly avoid having tmpfs complete for RAM in this kind of context. tmpfs use just makes avoiding "failed to reclaim memory" more difficult to avoid. (As various folks have run into despite having vastly more RAM than an RPi3B+.) So my /usr/local/etc/poudriere.conf has: USE_TMPFS=3Dno There are examples, like building rust, where anything but "no" or "data" leads to huge 10 GiByte+ tmpfs spaces for poudriere's build activity. Not a good match to an RPi3B+ . That is it for the recommendations of a starting point configuration. With such measures, I've been able to have poudriere with -j4 but also using ALLOW_MAKE_JOBS=3D without using the likes of MAKE_JOBS_NUMBER limiting it. (So the load average could be around 16 a fair amount of the time but still not get "failed to reclaim memory" kills.) Note: I'm not claiming above that -j4 is the best setting to use from, say, a elapsed time point of view for my poudriere bulk activity. END QUOTE > It started on the rockpro64 first, were i did a bit of fiddling, e.g. = en/disable swap, replace zfs with ufs, etc. nothing helped in the end. I = thought the board might be defective but now i start seeing the same = thing on the raspi as well. >=20 > Any ideas what this could be or how to debug this further? >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com