From nobody Sun Jul 11 05:09:50 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A63F412789E4 for ; Sun, 11 Jul 2021 05:09:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GMw1t2Kcvz4tFW for ; Sun, 11 Jul 2021 05:09:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625980192; bh=nsrAMUp7TYB1j80pkTJct1Hv8mQ0CsIT08ugQe36snM=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=ThRaqmlKIPZhSlG3WthzqC8UuCIWHhXNUiPGAbeVu8HuTn5c99KMNqeEQKXma7gtfPPzdfWtnkPv0PNv643+f1n/rYuh0167USr5NnZaj4cvvfu2QBPe/o0cL+RvKJvHkNcFJ17jVO4FyQhtLXj7ks4QSgYRsOzvsMhf3Sp8xJrBfbG+9ZN08LOtSdPisyj1S+s14XxhSXoLXnIdjt4wrB1q8oTA8g4Vr2iKKXAOgdM13kDmHPxmYqrF+/g2c1A5OYZF3NcrQHEhIGjI4J1htUy8HF8NnWxTwWM8ryQ5UHO1O1oQrVHgn/NQBm1APX2XvxOZToVYNaevVGtY5TgoMw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625980192; bh=mTONV0kda7xaCCfoDTq6D0aP8z5QVL8dQWkwHfEIez4=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=ZiS1NA6S8ELdBapoTKvjh60yI/RXNcH0uZNj0xdezmClF74NZiu5SH+90oo8cMVgKRgTo2C01eoNswympB3Kd/i/k0IBbdxSkvJe/qySUKWONyY/ccdMMoQznZoMEiZSyyfKGK1CqtXz1xgJBfVSV/F1paaC9NIZTAy+IsU+B1d89OruGRgPGJ6oAu5QedfckNw0g0lNpnq0PSu9mIACjD6e8cfuhmZiDyqHc31E8ZRG4qmJY9YVfYWS7xWTd0L2OM0cl4ddBsXaLeKGamwX403N3AuLd4qeWty4zUOV5JbEJZ4JGbvaBZrmTIjaehZDtIqCWS1YMANWGWtjygeQSQ== X-YMail-OSG: Exj4_asVM1m0lvegGvrUXi9vQm875sV5Nu6N65R3k5b6ROjP1Li_U3Li5AVPSq7 mcGLVWqOFI9.xFq9EVpNQH5yPrqUBYMVgw1PEssmnDdQJi8gKMbIJapr477w5LM3rekXiIPPbkDx MpyNrY75iDEMEaZSJEAN6WnDGzdrsISVF7xYnQiTEossB4uM2araXErEX519uHhos34Gj1RwqasJ r7aeg.alXV5UairESndtH04C8saiRDwBWBK15XGLoABx9aSdJYP27OdcN6kbMZ5A2F8JVM7OuC8O DOs7i4XZGHVNLIGR37NPKlUTy7X9OJ6tFwtZa3G3rsyIv790YCMsaQ5N89ZxCgMmSaM.yYSGXEup k0ftNe0xtBGdNmlJVehgpCgDTg9BR8wftVXG2HLtQhsxCmXXMqP1VNt0nVQMq2Xlb8OdJ1TSwDvy 7kfj.5jknjJ5hfiwFSdOAebRgljjE64jc77rujOHfNeCXPYIxm5xfzX59.bodonT2xxFP0DsStEF C2XC8wEU3CCcUUHw8laL32gnWQBVEv1eHcQNidVJBZ76CjEGGlBHz93JTMpbrqkZEZ1ca1FXFkNk Ci2N_nBvbJr5LKcNhWvXkoV5GcYkban_Sxxuh6kr4B.A3T5rJLCL5tGKHhskHhkdIkp8EZLQxMIh diZr0p8JyssPwnDfjv25.GghFKzvremEL1Q3heKelc7HOnl3J.NGs4n_LQIzdWWJ05Cx_cUbx4dP RHfUpg9YwZAG6QKsvzrPqaenySozHux.zXdwulcLxym572PIa_v.d0Wa4mmNzzf8M2x0eFqBRIo8 zVULe_hSHEExfAY2VZCyDEvy2ke38B6pT_AqLzJtNdngb3fbXx6gyIW9_4nDsP7tu1HiQJHNRjFD 7QKVMj8BAOQnbdGSASMVes97Mp2OKqg_tfSdmkbc.iwr1SJkWXdJKeTXTO01lch4yJsb9p5MgAvU Otx5pTTLkfZ5abi0Tb5JvNL_1zhefummQOUVv8784kpaf0ckCzOHIuyvUhFiZjxQz.W.R22WVG3E kQsL5Hb3xImOYc8h5uFSmJr2lV0TBaraEofOwjHV5dXCoGTh6JqYCmdqCTpbfbWKcW_NuxhAsPAr 7adPrhQnehq_W0J07yHTwsncNUJuhB4.ZsteoWaQj5UBYfmorAFdIcCnkKRHQDp8Cexgs3kCTnPG 7WrlmYbHgfpY41Ys6lOP1L89Kgodx3MWy_.Sw8G_a.1bV75TXbuk1_Qg66njNcdRKv5dbHOfpyoC jXc11sIbrrDWgpT.TfzCQdmrXc5Y1EStniAXlt8_zQ9qT1jR9NJkkwtt4npY8fJV3tYaQpjAu5mH 6Uz3Wi.wcXDq5HeIvvVUWxZHuKtEub7su4e4YvVRLAwmFi0aDxCV_SXlFyuGV5A_UvAnmkURHod0 OeloBtvT7kA1Z2NHIzeY87bC5vODjKb1f2iMVRXlkuosmy7J1dWwoGgTEHOd2YYgImpLpvxPRah. v9UiZmwR3c1pPUMCc86UxxdwUFpQoxlUHE0X_5HsxMJD2PvpTC394wA8Se28QnGZR.kd7VspiYNY wDBZNyuL3SrmpnQzHrW4i91gGjIJ4grpFrG3Uc9ukaBtQHV3maaPAbTqzunI8Nz8JXufju5jhUzr wkNd_cpeBP_c.f0f1D83pIW7tG_CnAb6c2tT6FM8h6ala.1bqELJn3.LUgVQ.z7SUahLAVAwZnZT jhJ0oErF9BIVaaNCU.TBsGVgOWuXFtz1jPGTjzFPc2viKMFYaej0qvlwR9BF.H8DcfXolM6dPOLS 6uvoJZSWlSOck3eVhJoNWT.2eoH2Ilq31rmGZxCj7x07z1AYX.KAzKD.6.2VXf0kxJgsaPPt395M N7chOnszWuT3CDXLlmkA12kEh7MeccEuiqVeDlpx4tIuqqoHnVR2t74eKCseVNUpgsC6QxTX4ZUi yoihNcQPDn.xl6qmazfqMnDrWGe3YSHGN7Hje4sDZmCWbPYgTcfM9A1vwl2E_MYbyp50AOqhMIdG EKYyrcvZlsF4SaHL0bZ5.n0G1HC5BKYVM23jBKs3cLk6PZHuwBZuA9KPgQUah14cQqjqa6rLV7le dQFXadne6kt_GJc.RD9F6VEh2ijKEdszomNWAgaglNGMn7SNqFPUN_Zfa8pGQdGuw_RAoB0FZJ4q 1G4ITOMTpfgzMAwVYVVokAmfTMcsSad6kxXTO.RflyGutWIy_tG_B3M_STJSIirkleH9wUR3maVW PGlayODhw2Wu2sk2k4uDkZEHh81TZ6K6OG.jbvfSF_boKeP9HJkUpNgERFSFWtHmISBaOrtGjt2E fWMzohdWuIDUTFkMdYVHGqmn5jnu7OdE5fCxk4kk1MDBRmAnE2Os.fPAw7mmCJRFUJcLYtS.oYMF 0v_0XKB3mxZUEmLIwYKyeOW1AYxeVTZHXmZz.BKgXN7MlhjS5tFAmLVFXECpz1mvyu43a4rAD X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sun, 11 Jul 2021 05:09:52 +0000 Received: by kubenode542.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID ad5929150b2ec595f6d8ce43fe2c3754; Sun, 11 Jul 2021 05:09:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: HoneyComb first-boot notes [a L3/L2/L1/RAM performance oddity] Date: Sat, 10 Jul 2021 22:09:50 -0700 References: <8A6C415F-A57B-4F2F-861F-052B487166D6.ref@yahoo.com> <8A6C415F-A57B-4F2F-861F-052B487166D6@yahoo.com> <40AE6447-77AF-4D0E-864F-AD52D9F3346F@yahoo.com> <12A4EDD1-A2AB-4CE3-AB0E-A4B5D6FB4674@yahoo.com> <5B1B5E1A-8AE4-4889-ABE6-50C206F896FB@yahoo.com> <7DBDC8AB-C80B-4E26-B58F-251A3D29CE41@yahoo.com> <5BBF1B55-F02C-4817-B805-677EDDC5B809@yahoo.com> To: freebsd-arm In-Reply-To: <5BBF1B55-F02C-4817-B805-677EDDC5B809@yahoo.com> Message-Id: <0B577668-97AB-44B6-B1A7-C68F6CC299E5@yahoo.com> X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GMw1t2Kcvz4tFW X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ThRaqmlK; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.30 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.148:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.80)[-0.802]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[98.137.69.148:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.148:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.148:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-arm X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jun-24, at 16:25, Mark Millard wrote: > On 2021-Jun-24, at 16:00, Mark Millard wrote: >=20 >> On 2021-Jun-24, at 13:39, Mark Millard wrote: >>=20 >>> Repeating here what I've reported on teh solidrun discord: >>>=20 >>> I decided to experiment with monitoring the temperatures reported >>> as things are. For the default heat-sink/fan and the 2 other fans >>> in the case, buildworld with load average 16.? for some time has >>> stayed with tz0 through tz6 reporting between 61.0degC and 66.0degC, >>> say about 20degC for ambiant. (tz7 and tz8 report 0.1C.) During >>> stages with lower load averages, the tz0..tz6 tempuratures back off >>> some. So it looks like my default context keeps the system >>> sufficiently cool for such use. >>>=20 >>> I'll note that the default heat-sink's fan is not operating at rates >>> that I hear it upstairs. I've heard the noisy mode from there during >>> early parts of booting for Fedora 34 server, for example. >>=20 >> So I updated my stable/13 source and built and installed >> the update, then did a rm -fr of the build directory >> tree context and started a from-scratch build. The >> build had: >>=20 >> SYSTEM_COMPILER: Determined that CC=3Dcc matches the source tree. = Not bootstrapping a cross-compiler. >> and: >> SYSTEM_LINKER: Determined that LD=3Dld matches the source tree. Not = bootstrapping a cross-linker. >>=20 >> as is my standard context for doing such "how long does >> it take" buildworld buildkernel testing. >>=20 >> On aarch64 I do not build for targeting non-arm architectures. >> This does save some time on the builds. >=20 > I should have mentioned that my builds are based on tuning > for the cortex-a72 via -mcpu=3Dcortex-a72 being used. This > was also true of the live system that was running, kernel > and world. >=20 >> The results for the HoneyComb configuration I'm using: >>=20 >> World build completed on Thu Jun 24 15:30:11 PDT 2021 >> World built in 3173 seconds, ncpu: 16, make -j16 >> Kernel build for GENERIC-NODBG-CA72 completed on Thu Jun 24 15:34:45 = PDT 2021 >> Kernel(s) GENERIC-NODBG-CA72 built in 274 seconds, ncpu: 16, make = -j16 >>=20 >> So World+Kernel took a a little under 1 hr to build (-j16). >>=20 >>=20 >>=20 >> Comparison/contrast to prior aarch64 systems that I've used >> for buildworld buildkernel . . . >>=20 >>=20 >> By contrast, the (now failed) OverDrive 1000's last timing >> was (building releng/13 instead of stable/13): >>=20 >> World build completed on Tue Apr 27 02:50:52 PDT 2021 >> World built in 12402 seconds, ncpu: 4, make -j4 >> Kernel build for GENERIC-NODBG-CA72 completed on Tue Apr 27 03:08:04 = PDT 2021 >> Kernel(s) GENERIC-NODBG-CA72 built in 1033 seconds, ncpu: 4, make = -j4 >>=20 >> So World+Kernel took a a little under 3.75 hrs to build (-j4). >>=20 >>=20 >> The MACCHIATObin Double Shot's last timing was >> (building a 13-CURRENT): >>=20 >> World build completed on Tue Jan 19 03:44:59 PST 2021 >> World built in 14902 seconds, ncpu: 4, make -j4 >> Kernel build for GENERIC-NODBG completed on Tue Jan 19 04:04:25 PST = 2021 >> Kernel(s) GENERIC-NODBG built in 1166 seconds, ncpu: 4, make -j4 >>=20 >> So World+Kernel took a little under 4.5 hrs to build (-j4). >>=20 >>=20 >> The RPi4B 8GiByte's last timing was >> ( arm_freq=3D2000, sdram_freq_min=3D3200, force_turbo=3D1, USB3 SSD >> building releng/13 ): >>=20 >> World build completed on Tue Apr 20 14:34:38 PDT 2021 >> World built in 22104 seconds, ncpu: 4, make -j4 >> Kernel build for GENERIC-NODBG completed on Tue Apr 20 15:03:24 PDT = 2021 >> Kernel(s) GENERIC-NODBG built in 1726 seconds, ncpu: 4, make -j4 >>=20 >> So World+Kernel took somewhat under 6 hrs 40 min to build. >=20 > The -mcpu=3Dcortex-a72 use note also applies to the OverDrive 1000, > MACCHIATObin Double Shot, and RPi4B 8 GiByte contexts. >=20 I've run into an issue where what FreeBSD calls cpu 0 has significantly different L3/L2/L1/RAM subsystem performance than all the other cores (cpu 0 being worse). Similarly for compared/contrasted to all 4 MACCHIATObin Double Shot cores. A plot with curves showing the issue is at: = https://github.com/markmi/acpphint/blob/master/acpphint_example_data/Honey= CombFreeBSDcpu0RAMAccessPerformanceIsOdd.png The dark red curves in the plot show the expected general shape for such and are for cpu 0. The lighter colored curves are the MACCHIATObin curves. The darker ones are the HoneyComb curves, where the L3/L2/L1 is relatively effective (other than cpu 0). My notes on Discord (so far) are . . . The curves are from my C++ variant of the old Hierarchical INTegration benchmark (historically abbreviated HINT). You can read the approximate size of a level of cache from=20 the x-axis for where the curve drops faster. So, right (most obvious) to left (least obvious): L3 8 MiByte, L2 1 MiByte (per core pair, as it turns out), L1 32 KiByte. The curves here are for single thread benchmark configurations with cpuset used to control which CPU is used. I first noticed this via odd performance variations in multithreading with more cores allowed than in use (so migrations to a variety of cpus over time). I explored all the CPUs (cores), not just what I plotted. Only the one gets the odd performing memory access structure in its curve. FYI: The FreeBSD boot is UEFI/ACPI based for both systems, not U-Boot based. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)