From nobody Sun May 05 09:15:18 2024 X-Original-To: freebsd-toolchain@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXJmP2757z5JSvV for ; Sun, 05 May 2024 09:15:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-19.consmr.mail.gq1.yahoo.com (sonic305-19.consmr.mail.gq1.yahoo.com [98.137.64.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VXJmM6j4tz4Gcv for ; Sun, 5 May 2024 09:15:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=BJQhMpc3; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.82 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1714900529; bh=htVInKxBHkGYiw42P/sDQG6Nv2dFSN8uQlCI1iqJYoY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=BJQhMpc3CrloaNTiFRfC8Rm8TawP93oIjZJ5bLAL/3CfFInDktVBOHn1PD2zylF2GqrXUjylTmPNbSs8IEh8L0DFwc68x25gALkBAW8fGY6HGTaQ3J9fN0RaeMJbJ3wPYBOr7e3Xioa10BorlmZJACD43tjwZ+omtuwuwB4NT6vOzXKSy1oCRbxtHN3Cqu081+6fDis3uwLKeog+h2tQ6jcY+LugIS6PhET0gFj1T7PAOFpXHps1J4/bzHOFt5l5uspRMAKUkKFISDDMhYADSFF1OIap4QnX5ZywqLl4E3Z3XLy3aaoNMSLULKnEisBtR7r4uvcKeksDcqGOUDMF5g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1714900529; bh=Or6kmZdC1NDU6I8krQJWzT5zbnwuTfxDXpq3rzj87RP=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=cBYZ31o9OvuxFe9XRbSCgxAX1hjpzLerNhtPLUyWXX1n4q6cPIPdGcUrRSY5U05kkUn8svhQoSVXE+aNG8dFKohq4i/S+EZ3HQYZ0WRFzzi0TE9QP/yxXTrzMvXL1s/2p2qJVXaOYFAfZ1r3Zvr/qVqLpiNBrjnqqotOD1zg1s8t0ZrjPMPmp8f9nUbjN80K/R8YlThv662EHBcwkSPLIm5FFIurnl/zYvZe/f2zNhGmZRWAb0To8lcB11x6tL932PAVJq3wixz7hjRPdmJvhZgrHJA9MJBVsoRCqCqIUkiHQ+dviGMeTcFkrGbfL0xHQoBPZ5FTselnYWS3xiyZqA== X-YMail-OSG: m0P2vxQVM1nFZqCT_CSltCwTLq4jgFv1O.7ORq12sbiVh.128JckZf_BvOwfQnt jZHzX_wcrMbl86fLepjM6vfRIRbEjEuJGEA3T51K5QqLBlDM0xMDM51.UIklGA4r5so4zNx5jxTn 7eNjnUgPUM6f0bZw7Jciw6zEQQe1I9oTO28t7w5O.THNwo1a_.KAAWfSc3_UEHb9kbW2w1DVW2La igN60FZ0yk_L32gsfsmx4F.IDArkUC2EBvJ1NqeYVAbi_gXDQd9E1MCxk9bzxoXE7iOhL.U0s2NU y5PDBIiFxIEktAEgEDozqtR0llEY2CEEQIOMVD8gd89DPl0oXwW83h8wRMdffiyCYskqyDA030WK K9LTxqbc2Tc.xvnGBk.55HRN4fWx3UFguMredOKCi_cXQ8qUBVGDO_XTQwraMFan1faBqU.Nzg9B reFgRnGcOYw1hNRii0SPcXffK7k23BXRn31vyP22ikzJSOAk7N1H61LN64DcW3qkRUJh5zQ3GepK BvuXdXimBSGMAgpjjuZqLsniDMZI6KLbzAsvUJe8czsSIzclQSbnGAMXhMmL.2wjNQZAw1e6EJs1 GBZXnSpremVxwAI1wuzRWCau9YdLsc5JsaG_f6JV8m0JDh42EAg_ljp6CKIc6SFnn5GdEJzGAn9C xKLMyQwK.odIP6iUHum9OLu9h.zp3mR1FOWlVDfVMTd71PW7MM3Sp6rufhzU2CeGW3Av_7.WijWx DRUEY11LNLFdjCfMjf1gK3S_PbKTIenrLJL_xvlig7uu57AYiVQNLUe99p5khyTYX9TkBuVvBnYw 6nFC3zc.L6lDO3mhVnptvJ_QDWd4v0evAWU5slJAEWT861kVwKs4b_mBpJQ7pWKAkVwTDGaTGchc AK1htAm..iwoURqfMgHcWAh45xkmrKm0aW.ytr.ejzR9Y6TIyVq5CqgexdauwLpTtOXtcih7gzC6 JrQgtmOoDIiVbkpmsMJdNMl2FWtjVcMl9EbYtKprtTxh3oXrnyxHYPhe_pXpHnh2JhxmVjs688co Jt0I6reuYBhXsES7GrNCcXjisMW6vWAojKtbpuuTzjRZjtk2Hy5.TYi9YpNO2h1.WEBV5vyLI3zL rSY_yVB0oHIvEzutQBxXt_36i8sQjW1PP2Et2ywPvIWnywgP0M_XRUhNjW48Z5.WIhpEF93R_BUh qzlAlhh71FGzq.E8EHSygUnRFkJiCZ38ewO80u3mbQWeQgBPbtT.MFJH78hWF_G3HhWHDwUlEBJV q6zjSk2cAZS9_1sJqX7A6iZjguHj5VhVLI_mtNPjEGMhG1FBjbSv21H2w9m8nrE2uDOSh3pK1PtY 9UPRYsxIY.q9t2Rc_qT_y7HKHPbswpLtfGDcyLkjwMKxbzP3VyX7XsvWbRSP4fL5Fh1G1Dha4QKk V3c_dzg8MD3IsP2rIosDx19P_0atVJESOmUmrxP12gj864cbsITGoXUzP3cgN17tgqO95kiP8PyJ mn1VFsatJaOYUTlgxwRj_BpsWSM_czojTrinFg5Cv.dVd0p2xVdct3bXXVMxXg60hm5GXssDjvzG 2v4ABUkLnYwoOdJClGKb_D4ckJ0oSzouaaK7i8oAkJAYZ37S8kP.oDOZi9lTjA1qT8PbC0CsJkoE VVJ80almlqQE1k0IheQFHYf3KAQVhKeVDXkVgMv20JGt4GkK6OPTBq5uAQb_MxD3o9RoAvhtUSVO FcA.VWRiW_6Ktg72qSsKkTUBsW1cgiml6qtFQT1aGyr.4jkqy_j0KFi9NjYFxTey5UBRI.EDseQf K5d3R7T5.oEVIseu2KqZd0D45kilLZ9RETMi2z54tyTFqvq2.05GcBYaTiqS9jG1DKkCxrjL1nQX HPNEJRxcdnHC4Ziyzr0byjLsZEE6PFGGY.TtH__HcEh9oBs6.vFfgZkDmV4di9PuMPHLTYuav2VF qx2vdMw4.Yhav2FypiSQtKe5.SbljqdK7MatGOQbjx1qigjg6WrIx9tRIVi_APq5HfgIVrOAggXB 9tIQVRYWhQWhyPZ8cgb2qk1n7EJWJTBDXkzZ2VecdD3n4jlkkICEofBihHbUzxgqI.Y5.qPoG6ud M4Cu.m2dYlIKIv6vHPArDMr24Z9tBGrdx4blbSe50WQ_Mact50XqACalWk5.rIhKX3TMszae3lbj srjvG8uzKDd5M6HsPoP2llLPwq.t8JKE7W8PkqKc_4Np.bBIizt9OXErWuR_zO8bfmrksFjfoodH mId_miC.EOnY6vNCxT7YVHoL5hLSgpo66mN7gZFnLrVbQ419zr59XR9yUXApkcftjpZlIoJzTMSH gJScXG_77Fg-- X-Sonic-MF: X-Sonic-ID: 42a02a04-4afb-4455-a73a-062495efc9ba Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 May 2024 09:15:29 +0000 Received: by hermes--production-gq1-59c575df44-xmm9l (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e49d87ee7fb5435f8b7313ea6fdbe830; Sun, 05 May 2024 09:15:29 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Maintenance of FreeBSD s integrated toolchain List-Archive: https://lists.freebsd.org/archives/freebsd-toolchain List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-toolchain@freebsd.org Sender: owner-freebsd-toolchain@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: I've started collecting tmpfs usage figures from a poudriere-devel bulk -a for later publishing some of the top ones (handy for TMPFS_BLACKLIST judgments) From: Mark Millard In-Reply-To: Date: Sun, 5 May 2024 02:15:18 -0700 Cc: Nuno Teixeira Content-Transfer-Encoding: quoted-printable Message-Id: <313298B4-803A-44A9-B0B6-6C5FF4644888@yahoo.com> References: <97E1D1BA-CBA1-42C6-8202-B6D654941C28@yahoo.com> <3BF173B4-E6A8-42B7-9A64-0E87E81B8597@yahoo.com> To: FreeBSD Mailing List , FreeBSD Toolchain X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.31 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_SPAM_SHORT(0.19)[0.188]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-toolchain@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.82:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.82:from] X-Rspamd-Queue-Id: 4VXJmM6j4tz4Gcv [Part of my hack's notation does not generalize well across various USE_TMPFS=3D alternatives.] On May 4, 2024, at 09:23, Mark Millard wrote: > On May 2, 2024, at 13:42, Mark Millard wrote: >=20 >> On Apr 30, 2024, at 19:08, Mark Millard wrote: >>=20 >>> On Apr 28, 2024, at 18:48, Mark Millard wrote: >>>=20 >>>> I've modified my local poudriere-devel to have Success and Failure = lines also >>>> report the tmpfs size at that point. Using, say, script to log the = output to >>>> a file allows later sorting and listing of the TMPFS usage = filgures. (The >>>> context is an amd64 one that has the resources to do a full bulk -a = with >>>> USE_TMPFS=3Dall invovled.) An example (from an in-progress bulk -a = that has a >>>> long way to go): >>>>=20 >>>> # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -n -r -k11 >>>>=20 >>>> [00:36:37] [18] [00:26:53] Finished print/texlive-docs | = texlive-docs-20230313: Success ending TMPFS: 10.67 GiB >>>> [01:04:41] [07] [00:06:58] Finished net-mgmt/telegraf | = telegraf-1.30.1_1: Success ending TMPFS: 10.52 GiB >>>> [01:03:32] [25] [00:06:09] Finished security/trivy | = trivy-0.50.1_1: Success ending TMPFS: 10.10 GiB >>>> . . . >>>> [01:15:56] [20] [00:00:54] Finished databases/pg_tileserv | = pg_tileserv-1.0.9_12: Failed: build TMPFS: 2.61 GiB >>>> . . . >>>>=20 >>>> Note that the design is for sort with -k11 to work for Success and >>>> for Failure. (This is why "ending" is in place for Success.) I >>>> choose to use poudriere -N (no coloring) for this kind of activity. >>>>=20 >>>> This helps for figuring out what all is appropriate for listing in >>>> TMPFS_BLACKLIST for a poudriere-devel configuration to avoid tmpfs >>>> competing too much for RAM+SWAP. (But approraite free file system >>>> space is needed.) >>>>=20 >>>>=20 >>>> "<" below is what is new, ">" is what was original, in >>>> /usr/local/share/poudriere/common.sh : >>>>=20 >>>> 5928,5934d5927 >>>> < tmpfs_at_end=3D"$(env BLOCKSIZE=3D512 df -t tmpfs \ >>>> < ${MASTERMNTROOT}/${MY_JOBID}/ \ >>>> < ${MASTERMNTROOT}/${MY_JOBID}/.p/ \ >>>> < ${MASTERMNTROOT}/${MY_JOBID}/usr/local/ \ >>>> < 2>/dev/null | tail -3 \ >>>> < | awk '{ tmpfs_use +=3D $3; } END { printf "%s %.2f %s", = "TMPFS:", tmpfs_use*512/(1024**3), "GiB" }')" Turns out that the above assignment fails for the likes of USE_TMPFS=3Ddata instead of USE_TMPFS=3Dall . The builder stops at that line and never executes the next line (such as when a job_msg line had been added). Inline replacements of the two ${tmpfs_at_end} instances seems to work okay for such. Something is special about the use of the assignment. >>>> 5942c5935 >>>> < "Success${COLOR_RESET} ending ${tmpfs_at_end}" >>>> --- >>>>> "Success" >>>> 5968c5961 >>>> < "Failed: ${COLOR_PHASE}${failed_phase}${COLOR_RESET} = ${tmpfs_at_end}" >>>> --- >>>>> "Failed: ${COLOR_PHASE}${failed_phase}" >>>>=20 >>>>=20 >>>> The form of use that I've done also involves (over?) use of >>>> MUTUALLY_EXCLUSIVE_BUILD_PACKAGES . It is not as good of data >>>> for this other use, but the same .txt file can be processed >>>> with: >>>>=20 >>>> # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -r -k3 | more >>>> [01:42:09] [04] [00:48:16] Finished lang/erlang-runtime21 | = erlang-runtime21-21.3.8.24_3: Success ending TMPFS: 1.92 GiB >>>> [01:38:39] [28] [00:44:41] Finished lang/erlang-runtime22 | = erlang-runtime22-22.3.4.27: Success ending TMPFS: 1.92 GiB >>>> [01:05:41] [02] [00:34:54] Finished lang/erlang-runtime26 | = erlang-runtime26-26.2.4: Success ending TMPFS: 2.02 GiB >>>> . . . >>>>=20 >>>> to find longer running package builds. This is subject to >>>> significant variation based on what other builders are >>>> running in parallel at the time and what sort of load >>>> averages are involved over period in question. The >>>> MUTUALLY_EXCLUSIVE_BUILD_PACKAGES that I've used will >>>> limit that to some extent. But the result is comparisons >>>> of some builds that have no activity in parallel by other >>>> builders vs. builds that have extensive parallel activity >>>> by other builders (of not-huge packages). >>>>=20 >>>> Note: In modern times the [1D: notation and the like for >>>> what the -k3 compares are not well placed in the overall >>>> list compared to the likes of, say, [20: . The day vs. >>>> hour comparison is not a straight forward thing to sort >>>> on. >>>>=20 >>>> Hopefully in a few days I'll be able to list off example >>>> top tmpfs usage for USE_TMPFS=3Dall and top build times as >>>> well (such as they are). >>>>=20 >>>> Note: >>>> This is from my personal environment. I've not tried to >>>> simulate how FreeBSD's official package builders are set >>>> up to operate. >>>=20 >>> I've decided that the "build times" list is not reasonable >>> for use: too much variability of context. >>>=20 >>> But here is a list of 148 or so of the bigger USE_TMPFS=3Dall >>> tmpfs-usage builds from a poudriere-devel "-N bulk -c -a" >>> run. It covers the > 7.66 GiByte tmpfs examples for how I >>> build. >>>=20 >>> # grep TMPFS: mmjnk-bulk-a-output.txt | sort -n -r -k11 | head -84 >>> [1D:10:04:32] [25] [02:14:20] Finished www/chromium | = chromium-124.0.6367.60: Success ending TMPFS: 31.76 GiB >> . . . >>> [1D:11:05:43] [04] [00:16:26] Finished astro/kstars | = kstars-3.6.6_1,1: Success ending TMPFS: 7.69 GiB >>> . . . >>=20 >> Note: I had not put ttk into the TMPFS_BLACKLIST as I had intended. >>=20 >>> For reference: Queued: 34535 Built: 33990 Failed: 159 Skipped: 100 = Ignored: 286 Fetched: 0 >>>=20 >>> After rebooting, I'm going to re-run a "-N bulk -c -a" based on >>> TMPFS_BLACKLIST having a hopefully good approximation to the >>> above list matching and TMPFS_BLACKLIST_TMPDIR also being >>> defined. Also, use of ALLOW_MAKE_JOBS=3Dyes but no use of >>> MUTUALLY_EXCLUSIVE_BUILD_PACKAGES : >>> RAM+SWAP: 192 GiBytes + 512 GiBytes =3D=3D 704 GiBytes. >>> Media: PCIe Optane 1.4T has the file system and the swap. >>> System: 7950X3D based. >>=20 >> For the 32 hardware thread, 32 builder context, with USE_TMPFS=3Dall >> --but also using the TMPFS_BLACKLIST-- the result was: >>=20 >> [1D:16:49:31] Stopping 32 builders >> . . . >> [main-amd64-bulk_a-default] [2024-05-01_02h27m06s] [committing] = Queued: 34535 Built: 33985 Failed: 162 Skipped: 101 Ignored: 287 = Fetched: 0 Tobuild: 0 Time: 1D:16:50:42 >>=20 >> So: a little under 41 hrs for a "-n bulk -c -a". >>=20 >> My adjusted version of top that keeps track of and reports some >> Maximum Observed figures (MaxObsYYY naming) reported Swap having >> 50674Mi MaxObsUsed. So SystemRAM+MaxObsSwapUsed of slightly >> under 241.5 GiBytyes. >>=20 >> So the system SystemRAM+SystemSWAP being 704 GiBytes is a little >> under 3 times the example observed usage, giving margin for >> larger needs over time. >>=20 >> Note the below build finish times and durations and that indicate >> the overlapping time frames for the parallel builds for: >>=20 >> [1D:08:08:33] [15] [05:25:13] Finished devel/electron27 | = electron27-27.3.11: Success ending TMPFS: 5.15 GiB >> [1D:08:24:30] [12] [05:37:16] Finished devel/electron29 | = electron29-29.3.1: Success ending TMPFS: 5.21 GiB >> [1D:08:49:26] [04] [05:42:10] Finished devel/electron25 | = electron25-25.9.8_3: Success ending TMPFS: 4.91 GiB >> [1D:11:53:32] [16] [07:50:11] Finished www/ungoogled-chromium | = ungoogled-chromium-123.0.6312.122: Success ending TMPFS: 8.08 GiB >> [1D:11:54:22] [20] [07:51:01] Finished www/iridium | = iridium-browser-2024.01.120_1: Success ending TMPFS: 6.65 GiB >> [1D:11:57:03] [07] [07:53:42] Finished www/chromium | = chromium-124.0.6367.60: Success ending TMPFS: 8.06 GiB >> [1D:12:01:01] [05] [06:49:56] Finished devel/electron28 | = electron28-28.3.1: Success ending TMPFS: 5.13 GiB >> [1D:13:01:04] [10] [07:23:00] Finished lang/dotnet | dotnet-8.0.0: = Success ending TMPFS: 4.65 GiB >>=20 >> But I'll note that all 32 builders were active over those periods >> as well, all using ALLOW_MAKE_JOBS=3Dyes without any use of the likes >> of MAKE_JOBS_NUMBER_LIMIT. It is a high load average style of >> building that basically almost always has work pending that can be >> immediately scheduled on any hardware thread that is no longer doing >> other work. Still, vastly less than 30hr+ for the 2 *chromium, 25hr+ >> for iridium, and 20hr+ for electron25. ( 27..29 are blacklisted on >> the build server for main-amd64 and there is no 26). >>=20 >> Reminder: prior experimentation had already fetched everything, >> thus the "Fetched: 0". Fetching would add to the elapsed time. >>=20 >> I make no claim that build servers should be pushed that hard on >> any sort of systained basis --but the above does indicate what a >> 7950X3D class 16-core/32-hwthread system can do for "bulk -c -a" >> activity: somewhat under 2 days for such a "bulk -c -a" to build >> 33985 packages (plus 162 failures). (The PCIe 1.4T Optane media >> helps as well.) >>=20 >> Based on the TMPFS_BLACKLIST that I used, the resulting largest >> tmpfs usage examples were (ttk had been omitted from = TMPFS_BLACKLIST): >>=20 >> [1D:13:21:23] [12] [00:14:48] Finished science/dynare | = dynare-5.4_7: Success ending TMPFS: 10.74 GiB >> [1D:12:55:42] [07] [00:40:11] Finished math/octave | octave-9.1.0: = Success ending TMPFS: 10.27 GiB >> [1D:16:34:13] [25] [00:05:04] Finished x11/kde5 | = kde5-5.27.11.23.08.5_2: Success ending TMPFS: 9.95 GiB >> [1D:15:29:52] [12] [00:09:15] Finished science/ttk | ttk-1.2.0_1: = Success ending TMPFS: 9.59 GiB >> [1D:16:30:49] [08] [00:02:04] Finished biology/biostar-tools | = biostar-tools-1.10.1_3: Success ending TMPFS: 9.56 GiB >> [1D:16:01:32] [32] [00:25:56] Finished science/code_saturne | = code_saturne-8.1.1: Success ending TMPFS: 9.50 GiB >> [1D:11:16:08] [15] [00:48:29] Finished emulators/virtualbox-ose | = virtualbox-ose-6.1.50_1: Success ending TMPFS: 9.48 GiB >> [1D:13:41:14] [32] [00:19:20] Finished deskutils/nextcloudclient | = nextcloudclient-3.12.3: Success ending TMPFS: 9.21 GiB >> [1D:12:14:10] [24] [00:16:20] Finished print/tex-xetex | = tex-xetex-0.99993_5: Success ending TMPFS: 9.15 GiB >> [1D:15:20:33] [18] [01:14:02] Finished science/paraview | = paraview-5.12.0_2: Success ending TMPFS: 9.14 GiB >> [1D:11:55:04] [32] [00:06:42] Finished print/tex-dvipdfmx | = tex-dvipdfmx-20230313_1: Success ending TMPFS: 9.14 GiB >> [1D:14:04:21] [20] [00:01:13] Finished print/texlive-full | = texlive-full-20230313: Success ending TMPFS: 9.10 GiB >> [1D:13:55:09] [20] [00:05:39] Finished science/libghemical | = libghemical-3.0.0_20: Success ending TMPFS: 8.98 GiB >> [1D:11:21:28] [02] [00:53:57] Finished print/tex-luatex | = tex-luatex-1.16.0_1: Success ending TMPFS: 8.92 GiB >> [1D:10:21:01] [31] [00:45:03] Finished = emulators/virtualbox-ose-legacy | virtualbox-ose-legacy-5.2.44_17: = Success ending TMPFS: 8.90 GiB >> [1D:14:26:10] [17] [01:13:36] Finished graphics/qgis-ltr | = qgis-ltr-3.34.6: Success ending TMPFS: 8.87 GiB >> [1D:14:20:59] [06] [01:08:24] Finished graphics/qgis | qgis-3.36.2: = Success ending TMPFS: 8.87 GiB >> [1D:16:13:36] [02] [00:41:13] Finished math/octave-forge | = octave-forge-20240403: Success ending TMPFS: 8.56 GiB >> . . . >>=20 >>=20 >>=20 >> An additional note for *chromium/iridium/electron* (and dotnet): >>=20 >> I also previously had done a "bulk -n -a" with those and >> more listed in MUTUALLY_EXCLUSIVE_BUILD_PACKAGES and no >> TMPFS_BLACKLIST use (but ALLOW_MAKE_JOBS=3Dyes in use): >>=20 >> [1D:01:50:13] [03] [00:33:53] Finished lang/dotnet | dotnet-8.0.0: = Success ending TMPFS: 21.23 GiB >> [1D:10:04:32] [25] [02:14:20] Finished www/chromium | = chromium-124.0.6367.60: Success ending TMPFS: 31.76 GiB >> [1D:12:02:49] [27] [01:57:58] Finished devel/electron25 | = electron25-25.9.8_3: Success ending TMPFS: 20.55 GiB >> [1D:13:51:47] [11] [01:48:15] Finished devel/electron27 | = electron27-27.3.11: Success ending TMPFS: 22.51 GiB >> [1D:14:53:51] [01] [00:57:16] Finished devel/electron28 | = electron28-28.3.1: Success ending TMPFS: 22.66 GiB >> [1D:15:51:46] [02] [00:57:52] Finished devel/electron29 | = electron29-29.3.1: Success ending TMPFS: 22.49 GiB >> [1D:17:19:08] [01] [01:14:40] Finished www/iridium | = iridium-browser-2024.01.120_1: Success ending TMPFS: 25.66 GiB >> [1D:22:36:10] [02] [01:09:31] Finished www/ungoogled-chromium | = ungoogled-chromium-123.0.6312.122: Success ending TMPFS: 21.43 GiB >=20 > For comparison/contrast: I tried it in my UFS based boot media on > the same system . . . >=20 > [I later give notes about graphics/pinta getting stuck, much like > print/miktex had if the ZFS experiment.] >=20 > For the 32 hardware thread, 32 builder context, with USE_TMPFS=3Dall > --but also using the TMPFS_BLACKLIST-- the result was: >=20 > [1D:16:42:48] Stopping 32 builders > . . . > [main-amd64-bulk_a-default] [2024-05-02_23h01m58s] [committing] = Queued: 34535 Built: 33990 Failed: 157 Skipped: 101 Ignored: 287 = Fetched: 0 Tobuild: 0 Time: 1D:16:43:59 >=20 > ZFS had been: Time: 1D:16:50:42 >=20 > My adjusted version of top that keeps track of and reports some > Maximum Observed figures (MaxObsYYY naming) reported Swap having > 45241Mi MaxObsUsed. So SystemRAM+MaxObsSwapUsed of slightly > under 236.2 GiBytyes. (The ZFS example had been: 241.5 GiBytyes.) >=20 > So the system SystemRAM+SystemSWAP being 704 GiBytes was a little > under 3 times the example observed usage in both cases, giving > margin for larger needs over time. >=20 >=20 >=20 > As for examples of builders getting stuck . . . >=20 > graphics/pinta in the UFS experiment had gotten stuck in threads > of /usr/local/bin/mono (mono-sgen): >=20 > [05] 15:31:47 graphics/pinta | pinta-1.7.1_4 = stage 15:28:31 2.30 GiB 0% 0% >=20 > # procstat -k -k 93415 > PID TID COMM TDNAME KSTACK = =20 > 93415 671706 mono-sgen - mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_lock_umutex+0x6a6 __umtx_op_wait_umutex+0x49 = sys__umtx_op+0x7e amd64_syscall+0x115 fast_syscall_common+0xf8=20 > 93415 678651 mono-sgen SGen worker mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 = sys__umtx_op+0x7e amd64_syscall+0x115 fast_syscall_common+0xf8=20 > 93415 678652 mono-sgen Finalizer mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd __umtx_op_sem2_wait+0x49a sys__umtx_op+0x7e = amd64_syscall+0x115 fast_syscall_common+0xf8=20 > 93415 678655 mono-sgen - mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 = sys__umtx_op+0x7e amd64_syscall+0x115 fast_syscall_common+0xf8=20 > 93415 678660 mono-sgen Thread Pool Wor mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_lock_umutex+0x6a6 __umtx_op_wait_umutex+0x49 = sys__umtx_op+0x7e amd64_syscall+0x115 fast_syscall_common+0xf8 >=20 > So I did a kill -9 93415 to let the bulk run complete. >=20 > I then removed my ADDITION of BROKEN to print/miktex that had gotten > stuck in the ZFS experiment and tried in the now tiny load average > context: bulk print/miktex graphics/pinta >=20 > They both worked just fine, not getting stuck (UFS context): >=20 > [00:00:50] [02] [00:00:25] Finished graphics/pinta | pinta-1.7.1_4: = Success ending TMPFS: 2.30 GiB > [00:14:11] [01] [00:13:47] Finished print/miktex | miktex-23.9_3: = Success ending TMPFS: 3.21 GiB >=20 > I'll note that the procstat for the stuck print/miketex in the > ZFS context had looked like: >=20 > # procstat -k -k 70121 > PID TID COMM TDNAME KSTACK = =20 > 70121 409420 miktex-ctangle - mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait+0x53 sys__umtx_op+0x7e = amd64_syscall+0x115 fast_syscall_common+0xf8=20 > 70121 646547 miktex-ctangle - mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = kqueue_scan+0x9f1 kqueue_kevent+0x13b kern_kevent_fp+0x4b = kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x115 = fast_syscall_common+0xf8=20 > 70121 646548 miktex-ctangle - mi_switch+0xba = sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1ae = umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 = sys__umtx_op+0x7e amd64_syscall+0x115 fast_syscall_common+0xf8 >=20 > It looks like there is some form failing race condition that can > occur --and does rarely occur in high load average contexts. =3D=3D=3D Mark Millard marklmi at yahoo.com