Re: 13.2 BETA2: how do debug META_MODE?
- Reply: Mark Millard : "Re: 13.2 BETA2: how do debug META_MODE?"
- In reply to: Mark Millard : "Re: 13.2 BETA2: how do debug META_MODE?"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Wed, 22 Feb 2023 06:04:05 UTC
On Feb 21, 2023, at 21:53, Mark Millard <marklmi@yahoo.com> wrote: > On Feb 21, 2023, at 20:51, Mark Millard <marklmi@yahoo.com> wrote: > >> On Feb 21, 2023, at 19:11, Peter <pmc@citylink.dinoex.sub.org> wrote: >> >>> On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote: >>> ! On Feb 21, 2023, at 18:10, Peter <pmc@citylink.dinoex.sub.org> wrote: >>> ! >>> ! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote: >>> ! > ! On Feb 21, 2023, at 04:55, Peter <pmc@citylink.dinoex.sub.org> wrote: >>> ! > ! >>> ! > ! > ! # cd /usr/src/ >>> ! > ! > ! # env WITH_META_MODE=yes make buildworld >>> ! > ! > ! # env WITH_META_MODE=yes make installworld >>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #0) >>> ! > ! > ! ## no more rebuilds below? >>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #1) >>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #2) >>> ! > ! > >>> ! > ! > But what is the difference between #0 and #1? >>> ! > ! >>> ! > ! awk, cp, ln, rm, sed, and many more from >>> ! > ! . . ./tmp/legacy/usr/sbin/have new dates >>> ! > ! for rebuilds after installworld (that targets >>> ! > ! the running system). Not true for #1 and #2. >>> ! > ! >>> ! > ! The dates on these tools being more recent than >>> ! > ! the files that they were involved in producing >>> ! > ! leads to rebuilding those files. That in turn >>> ! > ! leads to other files being rebuilt. >>> ! > ! >>> ! > ! make with -dM reports the likes of: >>> ! > ! >>> ! > ! file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target... >>> ! > ! >>> ! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/ >>> ! > ! was always part of the path for what I found. >>> ! > >>> ! > Mark, thanks a lot for the proper input at the right time! >>> ! > >>> ! > This put me on the right track and I mananged to analyze and >>> ! > understand what is actually happening. >>> ! > >>> ! > It looks like my issue does resolve itself somehow, and things >>> ! > start to behave as expected again after four builds. >>> ! >>> ! Intersting. >>> ! >>> ! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk >>> ! > ! ended up being newer than such a target and, so, causing a >>> ! > ! rebuild of that target. I was going the direction: that >>> ! > ! it is newer really is unlikely to justify the rebuild for >>> ! > ! the target(s) in question. The other direction about how >>> ! > ! it got to be newer is also relevant. >>> ! > >>> ! > I have now analyzed some parts of it. META_MODE typically finds some >>> ! > build-tools to rebuild, but then if the result is not different >>> ! > from what was there before, then "install" will not copy it to the >>> ! > bin-dir, and so the avalanche gets usually avoided. >>> ! > >>> ! >>> ! The implication is that "install -C" is in use, quoting the >>> ! man page: >>> ! >>> ! -C Copy the file. If the target file already exists and the files >>> ! are the same, then do not change the modification time of the >>> ! target. If the target's file flags and mode need not to be >>> ! changed, the target's inode change time is also unchanged. >>> ! >>> ! -c Copy the file. This is actually the default. The -c option is >>> ! only included for backwards compatibility. >>> ! >>> ! -C might have more of an effect in a reproducible-build >>> ! style build process than on a non-reproducible-build >>> ! style one. >>> >>> Yepp. "install -p" is used, see /usr/src/tools/install.sh > > That may be incorrect about what is happening for > _bootstap_tools_links and other things. Why do I > say that? Several points . . . I missed looking for a obvious type of evidence: -rw-r--r-- 1 root wheel 2355 Apr 28 15:20:53 2021 /usr/main-src/tools/install.sh The script is not executable, which explains the use of an explicit sh in: INSTALL="sh ${.CURDIR}/tools/install.sh" This tends to nail down that the likes of: install -o root -g wheel -m 555 . . . in the output is not an example of using the script. > I do not see "tools" in any PATH= so far, making implicit > use unlikely. > > /usr/main-src/share/mk/sys.mk:INSTALL ?= ${INSTALL_CMD:Uinstall} > /usr/main-src/share/mk/src.tools.mk:INSTALL_CMD?= install > > vs. > > /usr/main-src/Makefile: INSTALL="sh ${.CURDIR}/tools/install.sh" > /usr/main-src/Makefile.inc1:BMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \ > /usr/main-src/Makefile.inc1:KTMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \ > > Also: > > # kernel-tools stage > KTMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \ > > vs. > > # world stage > WMAKEENV= ${CROSSENV} \ > INSTALL="${INSTALL_CMD} -U" \ > and: > .if defined(DB_FROM_SRC) || defined(NO_ROOT) > IMAKE_INSTALL= INSTALL="${INSTALL_CMD} ${INSTALLFLAGS}" > > So: explicitly varying styles for various contexts. > > It looks to me like the make output would show: > > install . . . > vs. > sh . . ./tools/install.sh . . . > > based on if the script is not-in-use vs. is-in-use > (respectively). > > Looking at some of my logs I see the likes of: > > install -o root -g wheel -m 555 cap_mkdb /FBSDFSSDroot-mnt//usr/bin/cap_mkdb > > which looks to not be using the script and not using > either -C or -p as well. cap_mkdb is an example from > the _bootstap_tools_links list. > > Some of the below do have -C use, others do not. > > # grep -r '\-o.*-g.*-m' /usr/main-src/Makefile* /usr/main-src/share/ | more > /usr/main-src/share/sendmail/Makefile: ${INSTALL} -T package=${PACKAGE:Usendmail} ${TAGS_ARGS} -o ${BINOWN} -g ${BINGRP} -m 755 -d ${DDIR}/${dir} > /usr/main-src/share/sendmail/Makefile: ${INSTALL} -T package=${PACKAGE:Usendmail} ${TAGS_ARGS} -o ${BINOWN} -g ${BINGRP} -m 444 ${SENDMAIL_DIR}/${file} ${DDIR}/${file} > /usr/main-src/share/mk/bsd.info.mk: ${INSTALL} -o ${INFOOWN} -g ${INFOGRP} -m ${INFOMODE} \ > /usr/main-src/share/mk/bsd.info.mk: ${INSTALL} -o ${INFOOWN} -g ${INFOGRP} -m ${INFOMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dev} -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dev} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dev} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS} ${STRIP} -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dbg} -o ${LIBOWN} -g ${LIBGRP} -m ${DEBUGMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dev} -S -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dev} -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \ > /usr/main-src/share/mk/bsd.doc.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},docs} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ > /usr/main-src/share/mk/bsd.doc.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},docs} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ > /usr/main-src/share/mk/bsd.incs.mk: ${INSTALL} ${${group}TAG_ARGS} -C -o ${${group}OWN} -g ${${group}GRP} -m ${${group}MODE} \ > /usr/main-src/share/mk/bsd.incs.mk: ${INSTALL} ${${group}TAG_ARGS} -C -o ${${group}OWN} -g ${${group}GRP} -m ${${group}MODE} \ > /usr/main-src/share/mk/bsd.man.mk:MINSTALL?= ${INSTALL} ${TAG_ARGS} -o ${MANOWN} -g ${MANGRP} -m ${MANMODE} > /usr/main-src/share/mk/bsd.man.mk:MINSTALL?= ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},man} -o ${MANOWN} -g ${MANGRP} -m ${MANMODE} > /usr/main-src/share/mk/bsd.prog.mk: ${INSTALL} ${TAG_ARGS} ${STRIP} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ > /usr/main-src/share/mk/bsd.prog.mk: ${INSTALL} ${TAG_ARGS:D${TAG_ARGS},dbg} -o ${BINOWN} -g ${BINGRP} -m ${DEBUGMODE} \ > /usr/main-src/share/mk/bsd.own.mk:HRDLINK?= -l h -o ${_LINKOWN} -g ${_LINKGRP} -m ${_LINKMODE} > /usr/main-src/share/mk/bsd.own.mk:MANHRDLINK?= -l h -o ${MANOWN} -g ${MANGRP} -m ${MANMODE} > /usr/main-src/share/mk/bsd.own.mk:SYMLINK?= -l s -o ${_SYMLINKOWN} -g ${_SYMLINKGRP} -m ${_SYMLINKMODE} > /usr/main-src/share/mk/bsd.own.mk:LSYMLINK?= -l s -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} > /usr/main-src/share/mk/bsd.own.mk:RSYMLINK?= -l rs -o ${_SYMLINKOWN} -g ${_SYMLINKGRP} -m ${_SYMLINKMODE} > /usr/main-src/share/zoneinfo/Makefile: -o ${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \ > /usr/main-src/share/zoneinfo/Makefile: ${INSTALL} ${TAG_ARGS} -o ${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \ > /usr/main-src/share/zoneinfo/Makefile: ${INSTALL} ${TAG_ARGS} -o ${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \ > > Thus it looks to me like installworld produces a new > date for cap_mkdb and, in the material below, > ${WORLDTMP}/legacy/bin/cap_mkdb ends up having a > copy of that new date in the next buildworld . > >> The code for the _bootstap_tools_links uses "cp -pf", >> not install, to establish part of . . ./tmp/legacy/bin/ . >> (Note: . . ./tmp/legacy/sbin -> ../bin so is a via a >> symbolic link.) Before the "cp -pf" there is a "rm -f" >> deleting the target file before the copy: the prior >> file in . . ./tmp/legacy/bin/ is never directly >> preserved. (The new copy might still be identical >> to the old one: the source path one might happen to >> be identical as well.) >> >> # Link the tools that we need for building but don't need to bootstrap because >> # the host version is known to be compatible into ${WORLDTMP}/legacy >> # We do this before building any of the bootstrap tools in case they depend on >> # the presence of any of the links (e.g. as m4/lex/awk) >> ${_bt}-links: .PHONY >> >> .for _tool in ${_bootstrap_tools_links} >> ${_bt}-link-${_tool}: .PHONY >> @rm -f "${WORLDTMP}/legacy/bin/${_tool}"; \ >> source_path=`which ${_tool}`; \ >> if [ ! -e "$${source_path}" ] ; then \ >> echo "Cannot find host tool '${_tool}'"; false; \ >> fi; \ >> cp -pf "$${source_path}" "${WORLDTMP}/legacy/bin/${_tool}" >> ${_bt}-links: ${_bt}-link-${_tool} >> .endfor >> >> Note: This is for the !defined(BOOTSTRAP_ALL_TOOLS) case. >> Note: the code uses the abbreviation: _bt= _bootstrap-tools >> >> _bootstrap_tools_links is built mostly in terms of >> _basic_bootstrap_tools and _basic_bootstrap_tools_multilink from >> earlier logic. For reference, showing what ends up handled >> this way: >> >> # grep -r "_bootstrap_tools" /usr/main-src/Makefile* /usr/main-src/share/ | more >> /usr/main-src/Makefile.inc1:# _bootstrap_tools_links variable. >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=m4 lex >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=mtree >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=cat >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=crunchide >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=crunchgen >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=mkimg >> /usr/main-src/Makefile.inc1:_kerberos5_bootstrap_tools= \ >> /usr/main-src/Makefile.inc1:.ORDER: ${_kerberos5_bootstrap_tools:C/^/${_bt}-/g} >> /usr/main-src/Makefile.inc1:.for _tool in ${_kerberos5_bootstrap_tools} >> /usr/main-src/Makefile.inc1:# The tools listed in _basic_bootstrap_tools will generally not be >> /usr/main-src/Makefile.inc1:# case we use the _basic_bootstrap_tools_multilink variable which is a list of >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink=usr.bin/grep grep,egrep,fgrep >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=bin/test test,[ >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/cut bin/expr usr.bin/gencat usr.bin/join \ >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=usr.bin/awk awk,nawk >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/file2c >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=usr.bin/bintrans uuencode,uudecode >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/xargs >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/cap_mkdb >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.sbin/services_mkdb usr.sbin/pwd_mkdb >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/ldd >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=bin/chflags >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=sysctl >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=tools/build/cross-build/fake_chflags >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/mkfifo >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.bin/jot >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=sbin/md5 >> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=usr.sbin/tzsetup >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=${_basic_bootstrap_tools} >> /usr/main-src/Makefile.inc1:.for _subdir _links in ${_basic_bootstrap_tools_multilink} >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=${_subdir} >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=usr.bin/bmake >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=lib/libbz2 >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=lib/libz >> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=lib/libcrypt >> /usr/main-src/Makefile.inc1:# All tools in _basic_bootstrap_tools have the same name as the subdirectory >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=${_basic_bootstrap_tools:T} >> /usr/main-src/Makefile.inc1:.for _subdir _links in ${_basic_bootstrap_tools_multilink} >> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=${_links:S/,/ /g} >> /usr/main-src/Makefile.inc1:.for _tool in ${_bootstrap_tools_links} >> /usr/main-src/Makefile.inc1: ${_kerberos5_bootstrap_tools} \ >> /usr/main-src/Makefile.inc1: ${_other_bootstrap_tools} \ >> >> What the prior installworld did for the analogous files is >> a separate issue. > === Mark Millard marklmi at yahoo.com