From nobody Wed Feb 22 23:19:41 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMXDv3JYSz3sbBl for ; Wed, 22 Feb 2023 23:19:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMXDt1LqGz3pms for ; Wed, 22 Feb 2023 23:19:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=CWhSCYQu; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1677107996; bh=ZzGJvQjNGNcXZDCz3YmJgBhV0FyuXBjFUR+U3HWogPo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=CWhSCYQu4NfvUCSah/0Ae7NJ0z2BV292UBhwBrNoo640Z49/IUoFwdMmnR2EK1lv8J74hMTxzjqbuNvR38X07Gi+PXnJhyqgYkP8GfzN97fHRD6xOYzF/lEy7ixiq8E0QhKo7lnTRh2mQYSvhAtrvD9ekUBltcoNhVrFiinrS5ijDe7QT61hdWPydqRHkJyy/Yh19jwyLUjWChQfqteyI1KJOj2qrb4OycONRCEGKbwMv7wv4dbl9+AVGI/1OoVEcg3K1trVVqaNl289K19js3Ve4sTE5Hhue15WhVvtR3LR6ziWQDNCaY3pSoKjHDshsByvOVWolZyENr1r3+55EQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1677107996; bh=i1mrxuTa6GN0r1o2rUjx9KZBWySJtQfKyeemBLnjHI5=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=l9+ANSy2AJ2ott+5ek3+C4vgdG0Z6Qr1E9+RfD1lQrfaAEecbwTTRvDtQO8JTH6WfDlqkqBK6+LMY5H6HJlXT2adcE94cT99HsNT2vMChXXTVaXhas33HeywV6XTrFVNr6B9jMZnvwccj3HCIrm7k+vvvIXeS/Z8077A0aGTCduc5n1LqNZNO2ux8EE2YzqTXB0ZxXUjuhYBXuenHSKXCXh46RkfUbgU/fbjQ2N83/VZOonz1uDRQWJ6A6YEEEJS7c+FDEZ/ENiK+yXvZrobneLpc+6BGAMxXzP/FzFroh2Pxxne3Bxel0L3Ty0a9F/0gfLyzlAloAMIanOjDjFnLA== X-YMail-OSG: VtEsv2oVM1nay4jEJ71qpDeTaPN_AplIYL2.ZeCj32TmNNFlHvhK.h2bK8E7qAa VgA6mzFktlILwBJ8GLrjyUIujmc3MTTkBQ7h83AUhqrviFJKutHv8ecPbUVqrgrYrXuK3z9MFkuK 84gfXZ75.O1wNvga8DQrgFZ_UxIbR6g2CUz2w9QCEpHWCnCdat4kqhaNiwEcqoq5QXhaqXgK1iLH T23F.brZmWbOjbeihj_TlPuazk.6zNLeScopXTGF1Ua2Z6MdcolrmPdk2sJeoULQJz67pSYI5Kmo dDgGWACa_71vlb2BEAN59tlsKZFQxzToqO1GcfNW6cf8S9x.898cPBH6uT7zEQJ8Hx4OyaVEcN3i oz2FQzGXwLJBBiXhBIboym9VWHRmSNoEXlLcaW5nXKI74VGmAoa4cD0l84x5u9jqYDVwWqS2Bxid yyNzzkiw2RyuTc4EcuWS0RzvFShSufGRqQiXKPebxsaSuvtjKYM8TSp9d1EeCQNeMY1qmAImdl3I QsrFAu.7t37HqgAim.xBgAAoIJ_3m353I08FXlp0txeIrz20UYP3ZNOXmRtJXth46NV6xXJf_uN3 BKRIIdEm_o2HIvpbDbk2Lz0ZHp6mRD6JW9KXBzMglYJOfYJEgjC6iQ4EuV1q0jexDJiI0NvTbOL1 Li9fMIpQsgFey5i_iT.eG3H1BGrWGAwN7yQo_aNlNE4zbVjPLq5nZoyRIXUVSiDK22DALIYO_tBu 4IHtjIkBhDR1LudIZbL5q3dd9tQQJEBmd_yvDwnj6dYZWljuvO5NBMbYrOFD_jWXe8CnIa5SWk5E 2bwMjI0UAAHRJ3JoJnmvia0hP9UMfCXDr984UujZBmWuRFlJyXNC6Lc5nLlpxJTPuHyzC4we5jZF XMrWwW8PIngTu6IHnV7yenp2CUTuHL_kVml3ci6ic7xbRqyB_VxQdce1Q2oJleaYf0iNtlkerNMU MMtQU64BEOPCp3KlVCiDkvuHERLJ8eGcedyHHdZw4yRj4Sl1ZWzDzgJt74Mq6ZJNDQODoaiHq9Xj pyDU9CLWTawbEtOdkPvA76dqa.vPYxtTr_boEKpKXoViN5xMV38YxpzyYQp93QGbW31KMPB70Uz. qrHVWy9LAMG9Rr_a8KQafs2EMfZry8P0HDKnborhA5SarGeyYNwAUOWZh98AKcnUQaj7GRfiHNa5 lpplxw6MPC_H6O7q8Tfc608CU27aQ8eNNiR0IB7BYq4FCQKLB_FOa7VHBNky9flc9cuNCfZCQ_Df V835d0nX.xwgG7kthDjg6sXZngPrxX2OuX.5qa7iOekuoHFMBJ4KX_Y45u0Yk84sRIUd18nD.rmF 8ADfmh.D.wKIgKY5Le3xKRYg1ljXOAFUPh.05yK2iFx0tAM22tb3s5x6itVwB5MPAfuOi_wm_gf8 lKu66aiPzxcSK3R_M.c5COtOZvYdXtTgvC_v3e3CXUkIunxFnNOAjZ2M5CVNlc5K6FvjGIlcFkiE YlvhhUq8xKZBlm.O83Z420zWivAtuEZlwRcIab_tHMFUBLrUFGbz4YPet7nOJfmr4QX0XM3FODjr mpm3FmxJEw6xo4zmbU3NFExBUWpRLnvXE1FuQpDj1ZsLbWGpmV3J1q4nptF4VEjXKHsRXURxWOJV 0gOKUQfWbTrwO6W7MVcfBuqsgf3aMif2BbpUxyxx1o7IGyErVZkAydnZVtiLUBFVqHMTADIIMr0v fwynEedmAIjmG8L4G2Nr3mo.sT.FIy_s_86QsjWQZXTj4JNltiGC_Z_u8QZxIHQyuFkp_nXgMNQQ J8y0nClLExyXlFBMOKzghwmVEem4j8XZcvJdUgfhqDEX2bh_CF2CUXcW7TBNKMQpp6AKoTo2L94E 9epA5nU4CbOogOq4.M0CS1rbm79SGwf1kXoipRAmRtHug.gV0El4UcuSNWoYSWabPg9O9cQtL7a5 XSw0PMNVZ3sYxY3pLciez9e4rB33b869M6w15Wmvn4aTTUFam_.jeKzFRgIgQdhqgfHjFl5R.euc NzLmZOimhl7tKPKk7zoyPUnqg64ZF93M7WFmYIAggK_hd0ZQDQ7wxLxvpJrLD7HNK0jIibm3OB__ RK4VYqBQ5F8s4Zs4xekIub54GbveHLST1yMe0FJv5JsBlLM9SAnd3fo_1QlxMMwgtW0asiPGTaYE teXP7qrg9G1OuvcfRPtInXkjvbP9BVS2hidT8udPqJ0ydW6UyugA_w2lwsyMtR6vTMwMV52Nohaj GZ3s9gPVQayFKKKIBRjrDJqjQhPPTq4yre3SMIJr.SUk9vT0R96_fn_p4z0vWOWP5gmNs9I6mqQ- - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Feb 2023 23:19:56 +0000 Received: by hermes--production-gq1-655ddccc9-zvr4c (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 562c41f8babe2cce96acd2a118ed444a; Wed, 22 Feb 2023 23:19:52 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) From: Mark Millard In-Reply-To: Date: Wed, 22 Feb 2023 15:19:41 -0800 Cc: Bryan Drewery , Current FreeBSD , Peter Content-Transfer-Encoding: quoted-printable Message-Id: References: <3345EBA5-A09C-4E3F-B94D-39F57F56BDBB@yahoo.com> <73088.1611797582@kaos.jnpr.net> To: "Simon J. Gerraty" X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.981]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@FreeBSD.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from] X-Rspamd-Queue-Id: 4PMXDt1LqGz3pms X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N [Added a question about a possible typo in the old original message.] On Feb 21, 2023, at 13:31, Mark Millard wrote: > [This will be a question relative to the old material below.] >=20 > On Jan 27, 2021, at 17:33, Simon J. Gerraty wrote: >=20 >> Mark Millard via freebsd-current wrote: >>>>> Given an already built, installed and booted system version, I've >>>>> noted a big difference for META_MODE in 2 rebuild contexts (no >>>>> source updates involved): >>>>>=20 >>>>> *) Prior buildworld buildkernel, installkernel, installworld, boot >>>>> presumed before (A) and before (B) below. >>=20 >> Yes installworld is going to cause problems unless you tell meta mode = to >> ignore everything outside of the src/obj tree. >> But even that isn't enough as you note below. >>=20 >>>>> So I used make -dM for the commented buildworld buildkernel lines, = logging >>>>> the build output and later diff'ing them. >>>>>=20 >>>>> Result that I noticed? Lots of lines uniquely from (B)'s case, = ending with >>>>> one of: >>>>>=20 >>>>> file = '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/u= sr/sbin/awk' is newer than the target... >>=20 >> Yes. That would be expected. >> I think Bryan added some knobs to mitigate some of this. >>=20 >> grep -n META.*IGNORE share/mk/*mk >>=20 >> might be instructive. >>=20 >>>>> Many/most/all(?) of these seem to me to be unlikely to actually = need to >>>>> contribute to what needs to be rebuilt (just based on being = newer). So >>>>> the option to ignore (some of?) them could be useful in making = META_MODE >>>>> builds quicker. >>=20 >> Yes on all counts. That's why bmake provides a number of knobs to >> ignore such things. >> They are listed in the man page in increasing order of expense. >>=20 >> One of the risks of the sort of introspection meta mode does, is = delving >> too deep (like the dwarfs ;-) >>=20 >> The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to >> contain the potential insanity. >>=20 >>>> The following from one of the .meta files makes the point that rm = use >>>> in the example is unlikely to be important to needing to rebuild, >>>> despite it actually causing a file rebuild. Nor is the specific = echo >>>> command listed relevant. Only the "ar" command is: >>>>=20 >>>> # Meta data file = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/li= bc++.a.meta >>>> CMD @echo building static c++ library >>>> CMD @rm -f libc++.a >>>> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o >>=20 >>>> -- filemon acquired metadata -- >>>> # filemon version 5 >>>> # Target pid 22471 >>>> # Start 1611359217.214996 >>>> V 5 >>>> E 22961 /bin/sh >>>> R 22961 /etc/libmap.conf >>>> R 22961 /var/run/ld-elf.so.hints >>>> R 22961 /lib/libedit.so.7 >>>> R 22961 /lib/libc.so.7 >>>> R 22961 /lib/libncursesw.so.9 >>>> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE >>>> F 22961 22962 >>>> E 22962 = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/us= r/sbin/rm >>=20 >>>> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not >>>> actually relevant to if libc++.a needs to be rebuilt. >>=20 >> True.=20 >> If there is nothing under .../tmp/legacy that should be counted you = can >> just: >>=20 >> .MAKE.META_IGNORE_PATHS +=3D that path Was that supposed to be ("." vs. "_"): .MAKE.META.IGNORE_PATHS +=3D that path The later ones listed have ".". >> which would be the cheapest option. >>=20 >> If however there are things that should be considered you'd have to >> use a much more specific list of .MAKE.META_IGNORE_PATHS or (Same question applies to the above.) >> perhaps something like: >>=20 >> .MAKE.META.IGNORE_PATTERNS +=3D */rm >>=20 >> would ignore an rm binary regardless of where found. >>=20 >> worst case you might need something like: >>=20 >> # realpath >> .MAKE.META.IGNORE_FILTER +=3D tA >> # ignore anything here >> .MAKE.META.IGNORE_FILTER +=3D N*/tmp/legacy/usr/bin/* >>=20 >> Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to = match >> the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty = string >> paths to be ignored. >>=20 >>>> Of course, the structure also point out the judgment >>>> is specific to understanding the sequence of CMD's >>>> listed above. Only a hack of ignoring, not recording, >>>> or commenting out the filemon lines ending in >>>> /tmp/legacy/usr/sbin/rm would seem to avoid the @rm >>>> handling issue. Such might well have its own risks. >>=20 >> No hacking needed, there are a range of knobs to help. >>=20 >>> Just for reference for more about the sequencing involved: >>>=20 >>> Looks like in my example various . . ./tmp/legacy/. . ./*bin/ >>> actually are links to files in: >>>=20 >>> = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi= n/ >>>=20 >>> and the later after-install buildworld "Rebuilding the >>> temporary build tree" step leads to the updated dates for >>> files in that area, updated via the code that reports: >>>=20 >>> Linking host tools into = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi= n >>>=20 >>> So the prior installworld leaves updated dates and the >>> linking to those installed files makes >>> . . ./tmp/legacy/. . ./*bin/* have the newer dates show >>> up for the legacy paths as well. >>>=20 >>> In turn the dependency tracking via META_MODE uses the new >>> dates on . . ./tmp/legacy/. . ./*bin/* files to cause >>> rebuilds of more materials than if installworld had not >>> been done. >>>=20 >>> It is not obvious if Bryan D. would find the effort to avoid >>> this worthwhile for the contexts that drive FreeBSD's build >>> environment choices. >>=20 >> For people that use installworld and also want META_MODE >> it would seem some additional IGNORE work may be beneficial. >=20 > Since some of the paths reported ended up being links > (symbolic, as I remember), what are the principles for > which form of paths should be the basis for paths in > the likes of: >=20 > .MAKE.META_IGNORE_PATHS (The above may be a typo relative to .MAKE.META.IGNORE_PATHS .) > .MAKE.META.IGNORE_PATTERNS > .MAKE.META.IGNORE_FILTER >=20 > Target of link? Path to the link itself, not the > target? Both? >=20 > (Something had interrupted my explorations back then and > I'd forgotten about your note and have never done the > exploration. Trying to partially answer a question on the > lists lead to me reviewing the old thread and running into > your note again.) >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com