From nobody Mon Jul 12 01:29:51 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 280E58D49AF for ; Mon, 12 Jul 2021 01:29:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GNR5c5pw2z4nlJ for ; Mon, 12 Jul 2021 01:29:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626053394; bh=ShDYuPlyd9aozUgTXT3cLGRdOArvz9OKQ53/bkQ2MEA=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=iPAEPIEM+okVqJbMPO2PhNMwP/p0vsf/AHUgRFIazlIgq6KjhOaU8r+WcgORYM8YUJizgQUv/LA7ADvw6bzQAeqnurjwwzPXaCfDJtDMD8a6lc0ykggS20UrqbgFGMzvQieb/bUlKwa1O0rk8XSgqhpN4c3+gZEAoxoAT0Eb6GKy5j5eQaJhHRRP1rBFcn0UQ0cuMlfUG9Z1HtwWX5v3Rc5MRkHIBI222Bf3/iIFgsVAeN7VuXq3ln0v+GYPGrB2sc7w/G+GTbIkyd+E7FqD+e6fXaza0hiEdnMVjP1RQKToxBNgdNI3BXI4JhaYQCWX89JsjnkCKkkYnIQEF1dWAw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1626053394; bh=Dqa6gxNBFD9uapA31u5zgMccmpalszhfW6gqH2eiegX=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=TYdDm7uAakLPrkKt9RK3InOiLSsKThU8KYRDt95ybNOPu7H5kQW19posBXChLELkSC72DRdvPTwlOskCcZLhafODf7grf+eGh1W2vNbf0xVHkhgxjKXo9vKdtkjqlPORy+Yeg42ZUq5iMgrtiOXQf3h75fKaf1cvYnij9UuxmqmkJajjLNU90AslESGIBiVc/F6Qo8uG/KnXwj5B9UpcU3GDNt2zD47iwN68Pz0Gfq/ZvsJIOfRmxMTVcmDlV/jadzUJfN+TZhqxMbVUbY1FlztCl0PawkTN5HTOxhVWVMtHHcqdc/EoSqKl6vE2FCfX60MYDQM/e+UWioiwmuxczQ== X-YMail-OSG: cRztN2UVM1moq.D8DZMe3mEzWqDxkWWjGsL5HO2W7r8WjsFcwev5zn1i10S8HmA SHqEqFjvqgqb0M1jhuaUECq.DBHayzt2vU7Z8WF8kM9sycqFp2ROBFdh8DeQouATNe0.rYxgIjqP eJBO3zldWB3gVw8cBI7ruckdaM8.5L1BGP9q0seM.ueNcqZFwr3l79x3m9huloSIDCANoDiW0Owi fxIS0MiwnxNhwqUtFjAUtSUgAsBN1L4rlPTP352TcH3TYxNP0y7fB8gYDiHzkXmVkOeqLw1VHiQY 0qZsuplzbKFbCM6tyVz6SfrCEYi7f57VkB0_lDnGuoCCg9bJMo4hYh_bGJZ1RvqH_CXC0MeSPG3q h5x8QJt819uuH_jb5y1I4tFH52WKGypf5p9fhcQvTg1UkRU.xpnHG92jAk1ZL8kekx5No0Id4_bR tug312KA0IXp2k_B_fmqwj7S6loPCuLcs8x8otzm07GNnX3gJhHkLSl5znypAB7ww7mVEUdx0aJm eulmPCJGK2bJle2YsLBuZ8Hb46Zt6SgfbR72volarIelHRIrHFMsPJw.sLUClXQCs9GQouh3Kyu_ fedsg1KzUhX86YtbrQpb8lLXxlhdlk4CklgRXoMfOUtAwkpsJV8IgBa6T_yG2S4d503sFT5I5tUY DBptN7qB3oJNn7heUvsFe1rf9fxbMnzl0b71Q8oLCuB6KoxSOBVCzLAV6uoT_wHuJj3Sfa_HXglB awr7mAkuy3V_XuXZ6R6ZsvYNZ9tp.GlXiBRw1Qn5rHhpl.Mug_6jVJnyNAAcgCMwNEY0VvoF_HS8 tEV2hawaj5yMjtMh_NYGUHF0wiH1NPvPTLqhkHaugj_6ogXreUiw_qlrphL9xHYLW9sNxgCmNRxC 3_rcg1NxGitD_FMNpL2HFI7YNqn5l.xjYyowsdMP6.6D1F_xyKS6rQjG0YPo.0pH.cj65_5iVuqA 4dAwKVgHxD7Ex97.7FszwblI_mBpJteDpdYjQkR4vD4IfkR2U_AyKYLiSfcwVPRuQLIu_L2sj0zi IvSIE1I9gafq_OSUU0_pl6_wjleXDFOuUr7lQ.PRuBZJ7d5GEnDfudWvLnAjDf1e.fGTRrTe.53d SrYaBpTxAKVLWVZ_66cAxZKizdv7RqJ_bIWkKcAxAr3WiEZQdC20xeEzfWV8sfInzwpSeCq0bPNS p62A8jarJ4n9sKbrIRCEvmyHSpNoFjLZxVp3kg6SJbbrP4rFttbIQShC.bfomvPfzFMMyCZGs7DU 8gmQFt6JZZowp5QMHhjsOmQXDulBK2QVROvVC3hfQEm8RR6WrifI7OI57sVYMMb4pBEgpG77TQoF BYLU5rCzsJsIejNKfoMWFbMRDoEWYU_3.2I0Ypl8AhaM0RKccHn.Hz54MQXLMscrGu_jkpj3xMdK 6UcWRXhkzNh_zbJSDUVSz3YGqOp6VFp0YN3OVcBOfyGcW3nZT804dB_GMxd8Llk7Cp8KyzRbfioz cTx0oD8ae_1JBujphxK1AEvLUXY6LjC9lDcXtKs8Dx.O9R8bgQ19dxsxb_WOUog6KIlYzN9ckD79 KMS8WSWD_A0Uq04urTUI_66vb3hiAqBEZ_iFBuGHoFDqtxULjBMLkBBnpPf9byHWmSJGMGrFvVD9 qBqJO.SQ.mVeWPc.CEa15cnGx5jxZUDqCnC5sudzovGD084RnKE8a94fkD_7_DEsfdCevF8K.KZ0 grlrk23nyxPKaHBdHh.SlGablERFP5PTJUVfKzlUxgyVQhM1Mlb6l.sM9fGTEt2UrWejKoHb1A6k zM5o5cFqogYjrXm9tdikN8rKz0wnSUUN.nZBIFRie8fAVVu0o2whVM8a_7h5BZkkbVrK1iCvGE9J Wl3U._5tnTCL4bZq7cD5TkXLHEgd55gFEAo8jKhQxl4W6OkR4t4ADQQQzOyqnHQ6zJht22VrfxeC bg3UtRdhWQp5mUoZ8Kr0HovpWpyQKHjzCFcNKXSkMBVzOk19e6Hy8Zd_neVrU8q9KFzSTj2uDY9S 0nHM5oleHh.o3psbr269Z4V1eyPfKRLBp8tQISCEU5oxSfEfC6qLVCDQLT.kTbJEk_oWvSqNsi3r vzjUifNANraVHpI6SAh6mTq6o4kykqrO6iCwd75JwaAxr4mM9tdCT6.5TR3hZ2nf1QWPH2GgqZ.i 2ZP7oPvQiWVej9.g63Ckh4LDKK6ujy16hCGyBy2FFrYLhQIAZvEmt.k8HmLAGe2CXljknz5Fqt.a 7FK4oFL_qmyGDnnEblTg.WOx5yI.4fj5YGeQR0K8pscbdHUV2VCEr3MSTARgTItIITbWUGpLlKVp fFb5_gzNxkbyxFRaEjlrtltsiVpTOt8ObDLRISOFvZY3sJ4DSuLpuF1CLGnf4Ez3vDdb0D0E3myS FdfdrZmPbCNOkKRfcUTD0KukBram_8GzUP.y2i0wT6JPWK6DvPK2dZsNZNAEob.BwA_6mMucS.0I - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Mon, 12 Jul 2021 01:29:54 +0000 Received: by kubenode542.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 19987a45ef063654d8c02e9a1bf4274a; Mon, 12 Jul 2021 01:29:52 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: HoneyComb first-boot notes [a L3/L2/L1/RAM performance oddity] Date: Sun, 11 Jul 2021 18:29:51 -0700 References: <8A6C415F-A57B-4F2F-861F-052B487166D6.ref@yahoo.com> <8A6C415F-A57B-4F2F-861F-052B487166D6@yahoo.com> <40AE6447-77AF-4D0E-864F-AD52D9F3346F@yahoo.com> <12A4EDD1-A2AB-4CE3-AB0E-A4B5D6FB4674@yahoo.com> <5B1B5E1A-8AE4-4889-ABE6-50C206F896FB@yahoo.com> <7DBDC8AB-C80B-4E26-B58F-251A3D29CE41@yahoo.com> <5BBF1B55-F02C-4817-B805-677EDDC5B809@yahoo.com> <0B577668-97AB-44B6-B1A7-C68F6CC299E5@yahoo.com> To: freebsd-arm In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GNR5c5pw2z4nlJ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=iPAEPIEM; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.84:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[98.137.68.84:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.84:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-arm X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-11, at 04:03, Mark Millard wrote: > On 2021-Jul-10, at 22:09, Mark Millard wrote: >=20 >> On 2021-Jun-24, at 16:25, Mark Millard wrote: >>=20 >>> On 2021-Jun-24, at 16:00, Mark Millard wrote: >>>=20 >>>> On 2021-Jun-24, at 13:39, Mark Millard = wrote: >>>>=20 >>>>> Repeating here what I've reported on teh solidrun discord: >>>>>=20 >>>>> I decided to experiment with monitoring the temperatures reported >>>>> as things are. For the default heat-sink/fan and the 2 other fans >>>>> in the case, buildworld with load average 16.? for some time has >>>>> stayed with tz0 through tz6 reporting between 61.0degC and = 66.0degC, >>>>> say about 20degC for ambiant. (tz7 and tz8 report 0.1C.) During >>>>> stages with lower load averages, the tz0..tz6 tempuratures back = off >>>>> some. So it looks like my default context keeps the system >>>>> sufficiently cool for such use. >>>>>=20 >>>>> I'll note that the default heat-sink's fan is not operating at = rates >>>>> that I hear it upstairs. I've heard the noisy mode from there = during >>>>> early parts of booting for Fedora 34 server, for example. >>>>=20 >>>> So I updated my stable/13 source and built and installed >>>> the update, then did a rm -fr of the build directory >>>> tree context and started a from-scratch build. The >>>> build had: >>>>=20 >>>> SYSTEM_COMPILER: Determined that CC=3Dcc matches the source tree. = Not bootstrapping a cross-compiler. >>>> and: >>>> SYSTEM_LINKER: Determined that LD=3Dld matches the source tree. = Not bootstrapping a cross-linker. >>>>=20 >>>> as is my standard context for doing such "how long does >>>> it take" buildworld buildkernel testing. >>>>=20 >>>> On aarch64 I do not build for targeting non-arm architectures. >>>> This does save some time on the builds. >>>=20 >>> I should have mentioned that my builds are based on tuning >>> for the cortex-a72 via -mcpu=3Dcortex-a72 being used. This >>> was also true of the live system that was running, kernel >>> and world. >>>=20 >>>> The results for the HoneyComb configuration I'm using: >>>>=20 >>>> World build completed on Thu Jun 24 15:30:11 PDT 2021 >>>> World built in 3173 seconds, ncpu: 16, make -j16 >>>> Kernel build for GENERIC-NODBG-CA72 completed on Thu Jun 24 = 15:34:45 PDT 2021 >>>> Kernel(s) GENERIC-NODBG-CA72 built in 274 seconds, ncpu: 16, make = -j16 >>>>=20 >>>> So World+Kernel took a a little under 1 hr to build (-j16). >>>>=20 >>>>=20 >>>>=20 >>>> Comparison/contrast to prior aarch64 systems that I've used >>>> for buildworld buildkernel . . . >>>>=20 >>>>=20 >>>> By contrast, the (now failed) OverDrive 1000's last timing >>>> was (building releng/13 instead of stable/13): >>>>=20 >>>> World build completed on Tue Apr 27 02:50:52 PDT 2021 >>>> World built in 12402 seconds, ncpu: 4, make -j4 >>>> Kernel build for GENERIC-NODBG-CA72 completed on Tue Apr 27 = 03:08:04 PDT 2021 >>>> Kernel(s) GENERIC-NODBG-CA72 built in 1033 seconds, ncpu: 4, make = -j4 >>>>=20 >>>> So World+Kernel took a a little under 3.75 hrs to build (-j4). >>>>=20 >>>>=20 >>>> The MACCHIATObin Double Shot's last timing was >>>> (building a 13-CURRENT): >>>>=20 >>>> World build completed on Tue Jan 19 03:44:59 PST 2021 >>>> World built in 14902 seconds, ncpu: 4, make -j4 >>>> Kernel build for GENERIC-NODBG completed on Tue Jan 19 04:04:25 PST = 2021 >>>> Kernel(s) GENERIC-NODBG built in 1166 seconds, ncpu: 4, make -j4 >>>>=20 >>>> So World+Kernel took a little under 4.5 hrs to build (-j4). >>>>=20 >>>>=20 >>>> The RPi4B 8GiByte's last timing was >>>> ( arm_freq=3D2000, sdram_freq_min=3D3200, force_turbo=3D1, USB3 SSD >>>> building releng/13 ): >>>>=20 >>>> World build completed on Tue Apr 20 14:34:38 PDT 2021 >>>> World built in 22104 seconds, ncpu: 4, make -j4 >>>> Kernel build for GENERIC-NODBG completed on Tue Apr 20 15:03:24 PDT = 2021 >>>> Kernel(s) GENERIC-NODBG built in 1726 seconds, ncpu: 4, make -j4 >>>>=20 >>>> So World+Kernel took somewhat under 6 hrs 40 min to build. >>>=20 >>> The -mcpu=3Dcortex-a72 use note also applies to the OverDrive 1000, >>> MACCHIATObin Double Shot, and RPi4B 8 GiByte contexts. >>>=20 >>=20 >> I've run into an issue where what FreeBSD calls cpu 0 has >> significantly different L3/L2/L1/RAM subsystem performance >> than all the other cores (cpu 0 being worse). Similarly for >> compared/contrasted to all 4 MACCHIATObin Double Shot cores. >>=20 >> A plot with curves showing the issue is at: >>=20 >> = https://github.com/markmi/acpphint/blob/master/acpphint_example_data/Honey= CombFreeBSDcpu0RAMAccessPerformanceIsOdd.png >>=20 >> The dark red curves in the plot show the expected general >> shape for such and are for cpu 0. The lighter colored >> curves are the MACCHIATObin curves. The darker ones are >> the HoneyComb curves, where the L3/L2/L1 is relatively >> effective (other than cpu 0). >>=20 >> My notes on Discord (so far) are . . . >>=20 >> The curves are from my C++ variant of the old Hierarchical >> INTegration benchmark (historically abbreviated HINT). You >> can read the approximate size of a level of cache from=20 >> the x-axis for where the curve drops faster. So, right >> (most obvious) to left (least obvious): L3 8 MiByte, L2 1 >> MiByte (per core pair, as it turns out), L1 32 KiByte. >>=20 >> The curves here are for single thread benchmark >> configurations with cpuset used to control which CPU is >> used. I first noticed this via odd performance variations >> in multithreading with more cores allowed than in use (so >> migrations to a variety of cpus over time). >>=20 >> I explored all the CPUs (cores), not just what I plotted. >> Only the one gets the odd performing memory access >> structure in its curve. >>=20 >> FYI: The FreeBSD boot is UEFI/ACPI based for both systems, >> not U-Boot based. >>=20 >=20 > Jon Nettleton has replicated the memory access performance > issue on the one cpu via a different HoneyComb, running > some Linux kernel, using tinymembench as the benchmark. >=20 Jon reports that for HoneyCombs older and newer, EDK2's older and newer: All show the behavior on cpu 0. "[I]t may have always existed." Jon also reports that U-Boot based booting does not get the behavior. (I've never used U-Boot to boot the HoneyComb for any OS media that I've got around. In my U-Boot ignorance, my quick attempts failed for FreeBSD main and Fedora 34 Server media that I've been using with EDK2's UEFI/ACPI.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)