From nobody Wed Dec 14 08:41:13 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NX83Z2ZqMz4jpc5 for ; Wed, 14 Dec 2022 08:41:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NX83Y2BgQz3sx4 for ; Wed, 14 Dec 2022 08:41:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="dotONM/h"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1671007287; bh=wWtILOelNJISwd9agIQs4g8eQ0CVWsAVfWvBrkEnk0k=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=dotONM/hBTO0r7UPAmBfyILv0h0byKv4LlTzUztcSpJJQotCl1yDiNu0tZkRheKJnEHZYHgaMI0PB8kffojl/tTogXJ8RMpBQTSJrUMG+JUXBAWiKLE2JmvHPtUsdeOvMj0TPIor4rsdJnpg7aVzqeBeqOtS2ctzVxxx13xQq6/27rGFgpNkU9OKwa6X/MpJTBQFJ2Hgskbt0PfnLJ+J7dkNbSFIJbpvhOK6D/3JvP0NcV4aFe0VvMU+2HtsJV6G5Oe3ZKnY6d+6dvzQ+zAB19CAOU9WA8vC9lRKdzxlR2NZnOMC9asL5HAZJNV9TghfqgOk4w+K0u0nhi70tRi44Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1671007287; bh=vpC9hC9A73IkGKeCRg1O9KJkQqJVr9NBSz3expmWsBu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=sBEUpS4U4I75mL7QFiW6vpwrZoUF6ByK0UvB2lQEa5tOkDV868s35b+oE7tcS0MGhL1pLvF3gMO7HqqMd36wvbRU1rT4JVih9iQ5E8EeR0BsJGBEtVMRrGGsDbLNzH29CHNDN9m9IJ8rsDr+7QMiK5b0n9XmyVRlfZPQy+VTxawo0AH7UBRh/meDRTqeQIuy8vgnDAVPj1AFqFb2b5qLv/o18fVpMjK5uGS1MVCsMH2JoHpotkLxGZtCtg+a6ZjraSrAxzkvLzk4cfClLiKl7ZymvjfkY7XMVIrepRc1rEXSjwHpek2uCT2+471X7A1a5954DPCOEseAp36Rjq7amg== X-YMail-OSG: MNVIe5MVM1ndqDwsZghjhle3zNbp15VKVUwFGmrRMPtiU9t7yS0g_1T76fGNrre 7IvZrX7xsyR0GatgBKe.1jjfaQDfm42H31ziMYI8dFT4Rlx78XMO8lSOd.8R.weNPnzcd2QA7r1x m1UNdBAyzeE7ypn_cXsnKmcmH0prfFUGiE5EWsc8hc0Na_RnfMqAWjGRuSqEIYpFKwsBLRs5Bca2 3AI0at93qTZawZW1XnZlR0hxyVtNO0wl9_mbgdFzQErk8OrpQKVfnP0Np3Oi1AzccFqGmTrgzGle hL9mhGefWYI3fnPABxccijrMU8HOVZoGclEBuN8hKEMY3EZC.Yjk4fvaeeZ_iq0Dj.wMbBXiq7Hq IEFUlYkdImloK.a2qZQ5NBgAMnpQU56sey8VUPwTZLwZ28MGwZBtSUeJkoa_NoK37D8sq2V1lNNx iBPu2Rmuc93H0fiVRjC8Jkh7d.MxDwGJ6AwFN4k0Qtqs8DjX5k.RH4.pDsC34W3a9lzpAe0DEmH0 2HnV54iKpP3dAAqaw7x7mFOvzX142UvBDVN3v5Bnf5aKK7od6id5E8u05rLjYOijgowNYdIfnbsg xURdxEXwK2JXOpMh02CKVi_FcwYXdoxQTgTnm5MpDdqYOYpVm96Bi01o4hltXbMLUmbLNEYK4ePQ LLMav.drHj4ZVOkPuRq.g30baCqMghD40DWwD3V8CM7bK43Svic9aqZRS0Hl2bA5oyJ9afDx.llX MJBIX5H2NO_Uv6smfojo4bRSkEDk6CatuNXYfgfWq.w56PJAw1eybXkrRbkVIN1qPqk8zJeU8ir1 SdVbDi1QNPKcPhrNQ4FLBTmqrYKJRreqkWkN5PZDv4EHEp1sQAbKzGQlHAKy2QwHckX.QXdrs2RQ L_5XK.DqRDmFmZYRRYsrYPzZnpmyM8FLw7MEqPOYF9qedShAFXtE2VtU5zFo8S7LCdlkFU4IcmHV 52dBNAylFtqBMVMpCOln369e3EKOECfhtL6tSdjcKTNbTqUzDyfkRa_uw_qB9DRFwXrr4Ex2pK5a j4AL03yfDyqzHfBgiLq39byazOxPnVgSgisgAxD5R_K49leL.68FRaq_4p9ULDAzLnwX4tGucKX0 7pJB0kbcE27fObRa1A0iebyanrSt7Knvng_CSZJQlyRLUjjW7vJWJlzjNc_J_1qKuqOc7BXZKyFy 4sUCQVREZ_3an3nhilYZbv7H65Hswc6YGLbIvrjWHHDP.GYmLVI9uJSVwvif7ApU2IOLgcxKoFLg zE.NHFoZIXRAFM.Ym0dnSRfHwMa.M9NOQUzuKIfE2Vd39VHerJKe472tf7leYBqUE0mf6t7oIzxd 5QDqODOxQE3Zggaq1qCc9xG0KbNo3PhVq3BIqqaAGhW16GGJyLdFYBVQnCi4Pz5uWkqZv1GPyViS vYodERpNEoQyz7ShVwoM9Ihxu0ufarw44.LN_TFlQWGC6n4OK0iPeaQM_2ArWrCnyHV5no9JUFmt vOJFjIq3GAZetRNoRyQE3Ma85loK8CucdKbMof0c6J2hPhk2lxKwtthA0t0BXfzEt3f5rc.YfQvb EQlCP9YQpbWjz2leFBtnStPQs8NGFiww1Gtl_ZFiLYeiLjveaKgLkVtvnttFnbBScC6j_tuMf2E5 K_0ATRbv92E5A0X4sZ6XQkxcIFsbaAnXqBh1eWXaiT6_JG2KWbjzQp0bFR0vHAKq1mWAZGx_gQle .tcteEA9PKfZkng41ucz6LwCwzMi8l8hHdHXZPuSMKYk.EW3fLK7g1Yjs4XjSq4xzeqFDEAmvtu4 ILEclYdWeA932nbA6bB.I2hlURHRVC5t1Pw_tgA7WOxe7ynFV8KunM7LBB9DHXwPC.ZyuKvfJt56 KE1RvByGHxs4aF4RfzJvHUAw9SPe2Lw48vxfepwjqXLSwifMjNTLERmmzQwliB_3gEi0IAdbzUtN E24QxB55b9dIl7g4vibXgIFieu6ViSk8b6Ic9a.VmrT_xSo4oXkPPE1cqPWnEFvv4DbqQaO34vh4 TM1cCmrl1dMkxc9VIsDSjWozZDv0tFIzGKWy2W106WtErJZ_uCwfBxp6.tPUX0X7jL9H4fK530bb p1A3MG8lwKL9KwpgxFeTULwsIo9C1_yF1mk5SMQHwmxSApaShpc4uyQyJ0bwRKT3T1BTz_JxN8MY r2K3lX64izLHBzIXineQSiH3mLZ5V9kOlTjC0BoHt1ABQWDq9pdQFImoxYG4WjEP.V1S0EFPExiS jqAQVval9LKNHMP90b5IuupsES1.qXFKvjZQkILOeFtbjcVkm_S6idKKTfbPw5eeGDpUpLSOS5g- - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Wed, 14 Dec 2022 08:41:27 +0000 Received: by hermes--production-bf1-5458f64d4-97x65 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c3d783a96b9733a063d13dcb7e931fc8; Wed, 14 Dec 2022 08:41:25 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: Re: armv6/armv7/aarch64 vs. FreeBSD images (retitled from an arch reply, moving to freebsd-arm list) Date: Wed, 14 Dec 2022 00:41:13 -0800 References: To: jhs@berklix.com, freebsd-arm In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3731.200.110.1.12) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org] X-Rspamd-Queue-Id: 4NX83Y2BgQz3sx4 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Dec 13, 2022, at 22:45, Mark Millard wrote: > Julian H. Stacey wrote on > Date: Wed, 14 Dec 2022 02:22:10 UTC : > >> . . . > >> I previously had some kind of a 3B, & now have a "Pi 3 Model B+", >> but have not got beyond booting images, starting to customise /etc/ >> & then it crashes yet again, needing yet another dd, & repeat, >> Being certain one is using the right image would be nice. > > This does not sound normal at all. I expect something more > specific to your context is involved. > >> My notes: http://www.berklix.org/~jhs/pi/#images > > I'll look at the notes separately, possibly not tonight. > Note: I presume use of an aarch64 FreeBSD, not armv7 or armv6. Otherwise some things would be different below, such as via tighter constraints on swap space sizes for 1 GiByte of RAM. Quoting http://www.berklix.org/~jhs/pi/#images : QUOTE pid 17 (sh), jid @, uid 8, was killed: failed to reclain menory END QUOTE This is the FreeBSD kernel complaining about the configuration not well matching the RPi3B+ workload. In essence, it was unable to achieve it targeted minimum amount of free RAM in the sort of time frame (really: effort) it is configured for. Depending on what you do, the FreeBSD defaults do not work well for 1 GiByte of RAM. Swap space alone is insufficient because FreeBSD does not swap out processes that stay runnable. Just one process that stays runnable using a working set that is as large as the fits RAM for overall operation will lead to such "failed to reclaim memory" kills. But, if you are getting this, you will almost certainly need a non-trivial swap space anyway. I have a starting point to recommend, configuring some settings. As I've no detailed clue for your context, I'll just provide the general description. A) I recommend a swap space something like shown in the below (from gpart show output): => 40 1953525088 da0 GPT (932G) 40 532480 1 efi (260M) 532520 2008 - free - (1.0M) 534528 7340032 2 freebsd-swap (3.5G) . . . 67643392 1740636160 5 freebsd-ufs (830G) 1808279552 145245576 - free - (69G) This size (3.5 GiBytes or so) is somewhat below were FreeBSD starts to complain about potential mistuning from a large swap space, given the 1 GiByte of RAM. (I boot the same boot media on a variety of machines and have other swap partitions to match up with RAM sizes. But I omitted showing them.) It is easy to have things like buildworld or building ports end up with individual processes that are temporarily bigger than the 1 GiByte RAM. Getting multiple cores going can also lead to not fitting and needing to page. I'll note that I normally use USB3 NVMe media that also works with USB2 ports. My alternate is USB3 SSD media that works with USB2 ports. I avoid spinning rust and microsd cards. This limits what I can usefully comment on for some aspects of configuration related to the alternatives. B) /boot/loader.conf content: # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts= 3 #vm.pfault_oom_wait= 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) If use of vm.pfault_oom_attempts=-1 is going to be inappropriate, I do not have background with figuring out a good combination of settings for vm.pfault_oom_attempts and vm.pfault_oom_wait . I'll note that vm.pageout_oom_seq is not a time --more like how many insufficient tries to reclaim RAM happen in sequence before an OOM kill is started (effort). 120 is 10 times the default. While nothing disables such criteria, larger figures can be used if needed. (I've never had to but others have.) C) /etc/sysctl.conf content: # # Together this pair avoids swapping out the process kernel stacks. # This avoids one way for processes for interacting with the system # from ending up being hung-up. vm.swap_enabled=0 vm.swap_idle_enabled=0 D) I strictly avoid having tmpfs complete for RAM in this kind of context. tmpfs use just makes avoiding "failed to reclaim memory" more difficult to avoid. (As various folks have run into despite having vastly more RAM than an RPi3B+.) So my /usr/local/etc/poudriere.conf has: USE_TMPFS=no There are examples, like building rust, where anything but "no" or "data" leads to huge 10 GiByte+ tmpfs spaces for poudriere's build activity. Not a good match to an RPi3B+ . That is it for the recommendations of a starting point configuration. With such measures, I've been able to have poudriere with -j4 but also using ALLOW_MAKE_JOBS= without using the likes of MAKE_JOBS_NUMBER limiting it. (So the load average could be around 16 a fair amount of the time but still not get "failed to reclaim memory" kills.) Note: I'm not claiming above that -j4 is the best setting to use from, say, a elapsed time point of view for my poudriere bulk activity. I'll not volunteer anyone specific, but there are folks that have reported using RPi3B+'s on the lists, mostly via asking some question related to system operation. So you might be able to ask for information from anyone that really has an RPi3B+ in use, possibly describing the context that gives rise to the question(s). === Mark Millard marklmi at yahoo.com