From nobody Wed May 17 17:04:16 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QLzx018TPz4BMsg for ; Wed, 17 May 2023 17:04:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-24.consmr.mail.gq1.yahoo.com (sonic304-24.consmr.mail.gq1.yahoo.com [98.137.68.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QLzwz5xbyz44Cs for ; Wed, 17 May 2023 17:04:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684343073; bh=F1y0vyDXl47fqjOu8YqIASgHjNw1/2l5GHBCp7B9nwQ=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=I11R+5vQNzIdYujc0ElMBQehLFrkQaSDicL6ZIHXbmc9Uq6DwaJNgAD6iddP8srJDmsZR1KjU9zODPHiKlblKC+nl0+3sgosCiem+e8OmYYtitP7ODMBPRQSb9VrNBsbpf9zdH+iUj4CboVmiZiL1n+2IjDpoBoesJGH5HPWWI9aCeSbA5rsx4Ra5FWyMuK13Ngyc27gPPo84FQ3lAgPp84Pzmk8lSPDicwmXMSDRvADsPhmhCKplbqtn17jkklbqU5TeunARvhnlxQwW5LUYoynojtVEMGnCd4e4HHuXm8Oz16rmOuTPTGWT6iYOp5laqKyfsCwouphZ++siLJJ/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684343073; bh=hPlKGw0kocdgRX7BilcohBYlM6z6PJJpiNqczcSHz3A=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=THMBcwyzIKHXwoz5qlulw0Vu+rM4GHFYdeYkanZVl8mraJYoKxQM180n6UhkPnevk3VPqEAFSE/Cl91mxO7s9pfJfqg3zH0wRhLGTci9KbdMG9zqZuncRFh66lNisirU6j7Rl0gfBHj4sHIVKKvmB105vA0FEnMl6irrlgL2wmAdZ7CWD9FL79M6Fp68EPajgZjROCjA7QOkijCHOHkfFcDG7IPnxfneZHFPzpkWEH8BYtIJcmFaLhOkg95JIhsk/EZ3ptWeWOj5paFOwk2vTw4/DTLwsGDNUPG4B7rNH3cvmiQlRgN4uhhNKTSIuRK+3O2rh4nnJL9nES+HGW6MwQ== X-YMail-OSG: oDYVv74VM1mf_OkgguCzuYuaa7ot8C1V.cBij6OZc93vP9yNY8yLx0zC2QQuSLS _CIZ20VtVg1ZqMdxn7ZO7MNV9Vn7ss1hMNXQLYfU6pltBy3zYhI_nkG.FXhj84_9iqSryNhc8DbW CrhBcnL7rpFlzlF9TyX8M_BA0obrSUROO42hjQnvFuEZXwK6pFJyiWK21_.8pi7YMCXajzr.APWy hwa.U3h_uheSKe0fRwkqGH1IMra6sR3PV2TahDKK_21Mwtug66d8TBt_jsK0tTCvb7W_S8fA5nr8 DMBikKHubt8pI2qHlkQ4DFNv5cN6t4rtYKzG.GkzWE4XIpK4nnzvRgeV_2DPb2u8gRc0fYxTmWyF MCv8UE7DuATnI9YdUBQ_.btMBzsgu_5xqIsysMwhJ0cFNKfMxGR_bivrLUZ11gAAZR9iZ5md8_CS 1HGnmyVNEAGLtLEFCb1H2_0.6.4w769DZSgSpRd4F.ZFyrYNs.7QWZnF3VXgehhbfWtJOIeS8Ihp JTiYRQmZldU3SuWu3Mcxb3penXglPiNDUjzBkoEUT.pim24WIvgM6BDHXR.uZnHXKiRMkwV_Mgw0 q90YC24Iula0VTR3URK67aYeIzXyJHwZCObfoWoIU5k8.kWj0cjARoWYAPtPRjjQc3WSu2aeBHBS shOqs_r1Ff2x00BXFnwdgeN4H3vdRRa1EfCHXFIAiEDtuBlOSzOZ0CRafmPL_EKHro0quu49it6h RenEFBRTH6g3t0kspQgJcNG8oMeLPVR4a4S6VA3EovI7MA8ROZQc9irY_mqRWj3paBVnlO_m6roy g1mCX7ks_LqpPLWy1uvFCIR3lBPYBBbcI0zYceZr1BDchxed3BqVhVcWVrNUX5TT8gufuAlmw_w9 1bbx9iDsuPgJKZhoCq77ySIsyLbP7iTvbOT2Nx_5FAdoi_AwCaZtkjRUHyoVIZkiPF5SbyXVJd8t yHnAE7CahisHgq.B2rOqSHMifwrDi4q7lw3Ka_ftyx_R4RGEWLLJgkkRfQaoMJdzxnUsIIecZgQX 8jNxfZyaGmijtrnFeiMhTqjtMBRyD9xO3jnagaUZC2jS_5rgDcAJjuC2JP2E1r_RRXqQeMpjucBH Ok7D4ygtnVJbaiKpi5NJY_rvZUMZ51dRV9bMbkq_KNLhjQlsSj0nJ.pdI.wk8D80jU.MnfeqvmoD ICLRYJBJbtZkW0bF9IqTooWFnXNpVFXuiBJJf0LTZf9KyBngjNLPYI_UZ_3O2okJNhncaihcsBwe 7DlprLc.Bo5RmP_0HlOuIJ187FNCAG3Norf8bQVoslsLlPU3h8DTKZHLPW3FVYOOTgRuYuycbRCf mqyOxkSLA0tmmYwL4nsUyDQ6TqRIBPKJEJtNnj7uzGvn_IhaQ5q6Gdu.Dxjlk8CHbHtxwhHBZEjV bgJ21FujPDKb20WKyP0zspMyG7jHLd.mJoO3VoLEuUg4fPqG6jaWwPTfG4G7RnlXHhV1c8ss81F6 L.unbn3gEspi0Btmk2qZcHp98a3tbN_oZSkQUsAtctPuu28Cm4kMfCcMFR16.yvc8cx4rwJDf057 Tau5byJKflBb7reUNLGiaXHHiitPoqY.UvkqPvryp3OU.32S6SZVSDbcU8jn0LeTVeOuVdmbauAw In39zFexYHUHdqNbOSH4YDbz6aqAYzS4EFDqxTBGq2SaE93iBI1tMaU6hw21u5Ey6phgDvEBBFYD BilIzgPJW9sx0BmFCvK4ObZVO.CCVKLc7kYcjZiWpQLBXyGTZNtpVLoKJi9bEt66K32jpuWmztzS ZUvsGR5fRDKJjOlKW3UXpxFxk6xHBHviiiSDvpPFJM2fTs0Pb3CPgWcvm7pgRDnGFRZDE6MchYfh JDA5StObYgiQhZ3d8rD.QBD_73baZOhKigomyVIAMXNvH_UtbcwJSAEDPAODbLOT4CaSu9gSiqRE c3jxrUFaBXWn0J.UNSTFUMNNPl0c1rtc_chOsKYDD4.YucYvyJ2TlaKQKYo4ntDAgEjyRbQLR3Y8 zOpYIV2IXF97EAwr6WB1Sde5z7oHq4Z.dPaXucMsV3N6YycflMWJaF503W76b_0yTzhshLLtMlP1 xqrELmQNaTB5UZrBam9Cr2t61u5LGFBFsiWl2nYfLo0lyi6KSDcdt7rejxBoM4_aXqgEy3iODsy9 0P6_WEXOnLEqAbDMlHEvTtneFLk.IV6tVTGLEixAFP.j17Ef5WAxLcHDcNxT6wWhcnuRPt_rtRcI 0sf0wnxPANgVz9bTn082yc_.snissTCPvfCa64qKX50pI6JJodVESHtvSwoqkuohq6G6TmQmiVhh w X-Sonic-MF: X-Sonic-ID: 66ecbc69-6598-4e2e-919d-d66d563e4dd4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Wed, 17 May 2023 17:04:33 +0000 Received: by hermes--production-ne1-574d4b7954-6wwdb (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 4f0f2da37bb73d54220110d5d781a2c9; Wed, 17 May 2023 17:04:28 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Armv7 (rpi2) getting stuck in buildworld for -current From: Mark Millard In-Reply-To: Date: Wed, 17 May 2023 10:04:16 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <03F330A1-35E4-40D4-B9C6-407041BBEC58@yahoo.com> References: <7758885B-3115-47F0-A453-1C010313D4B8@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4QLzwz5xbyz44Cs X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On May 17, 2023, at 09:00, bob prohaska wrote: > On Sun, May 14, 2023 at 08:12:23PM -0700, Mark Millard wrote: >> >> I'm unsure if you have well avoided having any tmpfs based >> space or the like that would compete for RAM and use some >> of the RAM+SWAP. In the low RAM environments, I avoid such >> competition and use UFS to exclusion. >> > > No use of tmpfs, the line in /etc/fstab is commented out > I've commented out the changes to /boot/loader.conf related > to virtual memory as well. You mean all the names that start with "vm." that are assigned in loader.conf ? : #vm.pageout_oom_seq="4096" #vm.pfault_oom_attempts="120" #vm.pfault_oom_wait="20" I'd recommend at least: vm.pageout_oom_seq=120 just to make sure that kills are less likely than with the default value (12). The assignment only contributes to the choice of when to do kills, no other aspect of of the virtual/RAM memory handling. You might also consider: vm.pfault_oom_attempts=-1 I'll note that vm.pageout_oom_seq is both a tunable and writable. So, if you have assignments in multiple places, the most recent to execute is active. The same applies to vm.pfault_oom_wait and vm.pfault_oom_attempts . You do not mention /etc/sysctl.conf and its: vm.swap_enabled=0 vm.swap_idle_enabled=0 I'd keep those as well. > All were introduced in response > to slow flash storage, it looks like they're not needed with > mechanical disks and at least sometimes counterproductive. > Since these changes there have been no communications losses. Intersting. As far as I can tell the only two that might have contributed to that are/were: vm.pfault_oom_attempts="120" vm.pfault_oom_wait="20" >> I'll note that causing swap space thrashing can make builds >> take longer. "Thrashing" is not directly the space used but >> the frequency/backlog of swap space I/O. I always avoided >> configurations that thrashed for notable periods of time, >> via using -j given that I'd already avoied RAM+SWAP >> competition. But thrashing is also tied to the likes of >> spinning rust vs. various, for example, NVMe USB media. It >> is probably generally easier to make spinning rust thrash >> for notable periods. I'd also avoided spinning rust. > > I can't help but wonder if the dominant I/O bottleneck > on a Pi2 or Pi3 isn't the usb subsystem. There is not just one bottleneck. Spinning rust introduces latency on a scale large compared to USB2-bus latency contributions. For example, seek time for spinning rust. (More about this later.) The USB2 on the RPi[23]'s may well keep your spinning rust below its maximum bandwidth. But that is a separate type of bottleneck. > With none-too-fast > 5400 rpm mechanical disks there are no console warnings about > swap, despite obvious memory pressure (high swap use, high > idle percentage). Most of the time one thread is eventually > given elevated priority and the overload is worked through. > > This morning a Pi3 was found seemingly jammed. All four threads > were about 500MB in size, all had priority 20 with about 1% WCPU. > No console messages warned of swap pressure, but the system was stalled. > Occasionally one thread would get priority 21, but quickly reverted > to 20 so the jam didn't clear. After poking around interactively > reading man pages one thread got priority 135 and progress resumed. Just sounds to me like it was I/O bound thrashing, something that could make builds take longer than using a smaller -jN would do, depending on the details. Paging tends to be small transfers with random positioning, leading to lots of seek time for spinning rust. When thrashing the system can spend notably more elapsed time seeking than transferring data. A single paging transfer need not give enough context for a core to spend any notable time computing: it likely has to wait for more transfers to establish a big enough working set. Multiple thrashers tend to block each other from reaching the desired status. > For the moment it appears that, at least when using mechanical > disks, no adjustments to the VM configuration are needed on > either Pi2 or Pi3. Random user interaction via keyboard seems > helful to break priority ties when swap use becomes intense. I'd call the evidence for the "random user interaction via keyboard" inference weak. It is a kind of context in which it is difficult to have good evidence about what would have been different without the manual activity. I expect that if you had waited, the result would have been similar. But the evidence for that is weak as well. > Might it be possible for a script to detect thrashing and stimulate > similar behavior? I doubt that the utility vs. just using a smaller -jN for the build, leading to less time spent thrashing the spinning rust. === Mark Millard marklmi at yahoo.com