From nobody Mon Apr 29 01:48:35 2024 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VSR7j6Q6gz5HwVT for ; Mon, 29 Apr 2024 01:48:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-25.consmr.mail.gq1.yahoo.com (sonic312-25.consmr.mail.gq1.yahoo.com [98.137.69.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VSR7h64Wyz4kGl for ; Mon, 29 Apr 2024 01:48:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=p4Bj2Gsh; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1714355326; bh=kyIOUTqmwACEhjpLzDrmm2r+LwAdk6DQZyUCOi/v540=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=p4Bj2GshUOkiATquBOV23gpgYnzaCQVxOR0w75xBDeRoEWHYNiDhZnm/do3IgdZmVrSnPQZEwOtK4sCbTTZ38q5k3PyHrckRrbR08WMZdPXYtjjCdeDoAiCYUKf6jcFPtMga0nu3DcPNM92DuIg+FgxpeVowoB2ArCEXkaKLfCvgXReo9CNezILbnrlwrX9zbuX5XLEL4F7LhwNa1MI8mfHWSU6AfU52whpapPgtZ0g5DwCqG86Uew0p3G6DgVuqhDKpKnO+XVe7mEALBZR8Mq9CG52ngK83mtOQ7yWLuytZWTdg6My9+ueHGtCbCOA64kAFporK8cbFg0CDVNSl8w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1714355326; bh=SRHKyEEenNilVUxBJrXKPdNuo2mTeXwtDIywiaahE/d=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=seXoTK4iTBeWcgfskxKmeeqpST/9/jsWvt7ZmNBGr4eVW9Dy1pPYW0FiM7juLI8PJ1JEz/jDHAr1eWbUhCzZjn/MGqlEVi0jeUu0NrwjuHEGZmTPzboiSWMnkx/OXi8RyHaZjYvjZNHAN8k8RArD/iXsSmPzQRAvceWnIqEDoyIOpHOssiparYoGjp9YOTd7Nc9Y1yxduT8PF7HKNc/rbbGBXsDvJo2JQBXNXex1ihzdq0GqxZn4lTegq+uJ38h687d27PgOtAo7wle/wA1f7I+SqrQ/MvKxfvufbbv7CeyRWxvRJ1TI5KNeVYKKnq67KuH3WhhHCms7S+/ZBe17+Q== X-YMail-OSG: tXdLJIgVM1n1p30hL6QF6ATOtnXmOff6qHPOGzZfUz02.jfdS0Gj9dC0nGWUzH2 Pw41cD5UmJoKhbkPYdgvfXOytn78ERfhT4uqTA2l_N9W0pVeco3vGWhzvRXktd90jNxMGM1gyda0 Y9y82AV4ir.xhAEaPss0CnX7l9fmVGTr0Rfl2jg53F1zu3hAZd_R7L80bGDogBxW1H1pwpmvqojG VV7yBZqutbSKJDudc5osEGkD5mRNHe2ufiikCx877CBnxKmadOtUM4ahDxxA7XsFlEx5RxVDDNfm u493mSe8DPAEX2vE58b4ED_IklW2Q8EGIYNNzkTXkTVMYAD3gBfPqtmAxyI7ph2gJoEnMxY331Ty 5qXJDZaoUZHk_qRVEBymYCuo6KZQWrwToMumDAF3inmQrHE80HZjTq7etY9W73fBRPXBJ2_QHZOg R5DK3WGEU6VnKdGdRIx9SaotEZw.d75NfSCzxSortFsiUSLS_LgAH_KpZ2T2MZ87R7aQrACLafaT Vhc8Z6KEgBDTyChf.iu1sVtOJDZD41VXSGr_Eoffaf3NT1uEORUe5dDaQbVhkZ.Xz9nXp9NXPm.q B1QIBlQ1emWG7XUGOYPrmIfkl.ZmKa.uiOwgKjVSUbm9VcROEd.Nxiv5Z8wu4LGw_YeDl_6PARmq I8ZjhvGFsgjWXbFM_TfQPNy62kgouSshQxf_5ppqEUlvVlh4CrlXGIBCNgQzrONr_XWK7_hNENav Im2FrPGQPhY1Zj2IG_LNbBMQxFvyPPmTx4udZUuMt5Qxe5i8_7OfV5XXnUD5vDZGBztt1pKwAt4a KsJhTIPXFb_ZjQ4.OEvlSlVaB1LpTs9FbMVxnxNyoYoiQGgctTjj4cjYTBCSdU7zBGdo4EhYIcs8 W8mJIvKtECgg1YHHZLgpuo3XSReh8qVEOs9fLfyGMUWWnye1313ptmFdgADrHPPJmYqbQV1_04Id f0ZP5Un6PNphPlNh0E87nsfFxNJjidpjMRjyoCE6Ct4GYAnuNexZ67hM5juPOZnZkdJQoIeQ5mJR VsEaeqWLFZwTR.OT80be76HH_QsywSR7Z6.faqLdEQRA.f9yr8UaVhZuPbKLRb3TWYAQ.Rks7QJ3 VmraZRzYo.xKpm5DeJ6AuGGfVLHGOUdwFoPRO1psrLjFNFSoJD1654Y7p0sNzNboSIQBuMDST14v LkKdfQACk5ZsD0qnv0Im8jcoZUKuomCvKun.0e3BUUW0SdWumtks7V_FTaaOAess6tErz_V59hNk KLmGGZlK8TiiEJ85v6M0KLfQYw_U0O646A5m4ZG.0kxn8tHv7heuW2j8AY2e0hpMtqDZ7DjQYEiX 1kANNe4C69OM4ZPfdI4xkfvLGmgZvS9h5N9bHO0gYwPrk9o9ky8dgAefcWhpxE3jTq7SZTofWSJD pt5FVS0so.TAEeQtyk.jV3hnoIJkHWFveUkXmemtW3Su1eYaBR_oPWqhmTHFVQ2Liu0gjvs6fxYg ul9cgUiTkx0m.v0G0sgSO0SOMW7UdZwsijc9aB9.cqIXzIOumTkHBV4TPWbOj.E7xcNUEsNz_wf8 TlZuyEsGQeVw.5JjcNmmgieSMRbnoI.Rh99.lg10XTfh57Kkf3_81qDr0rCTjynoYUAVzqGn2MKm rPPtjibR0n33OH5FCqq2uCGfpwm9SaSv1OspzpNUEpIbPDokc4Ikg3_wyrjLvSG93lY7fvbbYJmu syHE074SAKsd07bvoNFPoord1yUKeaiuPWFvHLXheEPLVDI01Z30wJHNO35cpZVYRFP7fMi33Oq. xKJXaA1MiihWkppLLbeEgK8yjAd2kuXsiJN71Z50YgNSyT7fORc1jBNDjQMYDmCvmDj8C_3e8uw7 SKKBbSwGAtrmz6PpSdEKE8RxMK5y5Jc7zH549Sk7ABAtD9MTdewuKWOQKSBZTnvmwpaNTOsKur3P Ck0uyC5512IB4LceVEZd.bBaQYvnjPb61nfL2otrd7t2eQRXjE6x3146OfHpuzMG5W6P9yknEUhM HAIrsJB71LoUI92vyn1z4e6Qf6cuqtQzgCQxg8T1JtfINSdvUVcgzkh74Fv2As3ynBpRYQUUXVLK HeJ61KqygieuilQOmOGiQO4egy8hx_y2095gCK.gGS6zsoYTP70YKYbfBkF529.h_OPLpfE0S126 bz3NGTe5RX6q3fJiLV3JRnwGCU.FymOX3anlMnUJYdedHCdLv6Llbvisf9SfnhuzUdeGUz3YuIec BA9L8L8N8bJAzXZAUmUetN5drhzGpjtccVfIvrUOnbOrdsbWCK2kTpUfd3er3I1E1kthtKYFQHeS K X-Sonic-MF: X-Sonic-ID: 5fe745b5-9f64-4eb8-8915-215b327e459b Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 29 Apr 2024 01:48:46 +0000 Received: by hermes--production-gq1-59c575df44-qrd65 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID ca1051fbf2b49e58002df789a15edfce; Mon, 29 Apr 2024 01:48:46 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-ports@freebsd.org Sender: owner-freebsd-ports@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: I've started collecting tmpfs usage figures from a poudriere-devel bulk -a for later publishing some of the top ones (handy for TMPFS_BLACKLIST judgments) Message-Id: <97E1D1BA-CBA1-42C6-8202-B6D654941C28@yahoo.com> Date: Sun, 28 Apr 2024 18:48:35 -0700 To: FreeBSD Mailing List , FreeBSD Toolchain X-Mailer: Apple Mail (2.3774.500.171.1.1) References: <97E1D1BA-CBA1-42C6-8202-B6D654941C28.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-ports@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.206:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.206:from] X-Rspamd-Queue-Id: 4VSR7h64Wyz4kGl I've modified my local poudriere-devel to have Success and Failure lines = also report the tmpfs size at that point. Using, say, script to log the = output to a file allows later sorting and listing of the TMPFS usage filgures. = (The context is an amd64 one that has the resources to do a full bulk -a with USE_TMPFS=3Dall invovled.) An example (from an in-progress bulk -a that = has a long way to go): # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -n -r -k11 [00:36:37] [18] [00:26:53] Finished print/texlive-docs | = texlive-docs-20230313: Success ending TMPFS: 10.67 GiB [01:04:41] [07] [00:06:58] Finished net-mgmt/telegraf | = telegraf-1.30.1_1: Success ending TMPFS: 10.52 GiB [01:03:32] [25] [00:06:09] Finished security/trivy | trivy-0.50.1_1: = Success ending TMPFS: 10.10 GiB . . . [01:15:56] [20] [00:00:54] Finished databases/pg_tileserv | = pg_tileserv-1.0.9_12: Failed: build TMPFS: 2.61 GiB . . . Note that the design is for sort with -k11 to work for Success and for Failure. (This is why "ending" is in place for Success.) I choose to use poudriere -N (no coloring) for this kind of activity. This helps for figuring out what all is appropriate for listing in TMPFS_BLACKLIST for a poudriere-devel configuration to avoid tmpfs competing too much for RAM+SWAP. (But approraite free file system space is needed.) "<" below is what is new, ">" is what was original, in /usr/local/share/poudriere/common.sh : 5928,5934d5927 < tmpfs_at_end=3D"$(env BLOCKSIZE=3D512 df -t tmpfs \ < ${MASTERMNTROOT}/${MY_JOBID}/ \ < ${MASTERMNTROOT}/${MY_JOBID}/.p/ \ < ${MASTERMNTROOT}/${MY_JOBID}/usr/local/ \ < 2>/dev/null | tail -3 \ < | awk '{ tmpfs_use +=3D $3; } END { printf "%s %.2f %s", "TMPFS:", = tmpfs_use*512/(1024**3), "GiB" }')" <=20 5942c5935 < "Success${COLOR_RESET} ending ${tmpfs_at_end}" --- > "Success" 5968c5961 < "Failed: ${COLOR_PHASE}${failed_phase}${COLOR_RESET} = ${tmpfs_at_end}" --- > "Failed: ${COLOR_PHASE}${failed_phase}" The form of use that I've done also involves (over?) use of MUTUALLY_EXCLUSIVE_BUILD_PACKAGES . It is not as good of data for this other use, but the same .txt file can be processed with: # grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -r -k3 | more [01:42:09] [04] [00:48:16] Finished lang/erlang-runtime21 | = erlang-runtime21-21.3.8.24_3: Success ending TMPFS: 1.92 GiB [01:38:39] [28] [00:44:41] Finished lang/erlang-runtime22 | = erlang-runtime22-22.3.4.27: Success ending TMPFS: 1.92 GiB [01:05:41] [02] [00:34:54] Finished lang/erlang-runtime26 | = erlang-runtime26-26.2.4: Success ending TMPFS: 2.02 GiB . . . to find longer running package builds. This is subject to significant variation based on what other builders are running in parallel at the time and what sort of load averages are involved over period in question. The MUTUALLY_EXCLUSIVE_BUILD_PACKAGES that I've used will limit that to some extent. But the result is comparisons of some builds that have no activity in parallel by other builders vs. builds that have extensive parallel activity by other builders (of not-huge packages). Note: In modern times the [1D: notation and the like for what the -k3 compares are not well placed in the overall list compared to the likes of, say, [20: . The day vs. hour comparison is not a straight forward thing to sort on. Hopefully in a few days I'll be able to list off example top tmpfs usage for USE_TMPFS=3Dall and top build times as well (such as they are). Note: This is from my personal environment. I've not tried to simulate how FreeBSD's official package builders are set up to operate. =3D=3D=3D Mark Millard marklmi at yahoo.com