From nobody Thu Aug 31 12:45:41 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rc19k2cm5z4s875; Thu, 31 Aug 2023 12:46:02 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rc19k1tndz3Rwy; Thu, 31 Aug 2023 12:46:02 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693485962; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gJIp1KXlbbu15+zCuz2IsPc8SIsmWjaPZi7Vj5V/4JQ=; b=MIuo7aRcybFH1/eiKjYXbwQvDyXf0juvLbuIELD2bvAC9pqOiZKepmRMlDP3lpef4STTmf mzsOAoOqttardW4tp5mA405ysskw6frhLVqhChvZEC23quuCGUWk74AsYkruFGaLhMZEWC hmcpmn9/Q7zyxitVh6EfgfgIEMXVKivyscPc9jQ/uy3OIPDgIr3do7TLq7NND4pMJIcpY5 /ktfAJDKSYAMEfaa9GAz2LRLqx2WWPeW2CZihxYmsvyZ7xi6eKyAVzNghC5LHifeBfCXlX P3HkVDhCIEUAVPQF1PNDnhU3gzvWgttbZaB7qMJRiP1KqvMw9kzVKbG08I/B5w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693485962; a=rsa-sha256; cv=none; b=imiP4BE44VcdZjr90NSMcWwPyYDoJFz5zOxKDC6sw96O5pn9bg9OkwedEMjcjWL8f34bsK JWk7i4o2X017+bsvGhOCK/oR5c6gUfuQMK6+q23UD6ygcWyaaQrDz2Pv2Q64C3hkqOhHKw hd+I8HPjQNF9knsF0/b1nj8yXh/1WN6qweC8HE1at70s9SxQbv7F7ti3nS02TdR7Wf5A8+ ktAEcZxeXWG68ChfZc0g8cV+7GmdPXAryUIrzpbcQTx+8Z6lt2D8/jJTgQLoIbuSfuYdFJ 1Nr2Pdk5nIZXPyCuCKPi8RZ9SXm7lGcyborE6tS+MHIhVoPk+fU6eawAV2IA5w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693485962; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gJIp1KXlbbu15+zCuz2IsPc8SIsmWjaPZi7Vj5V/4JQ=; b=ho4CfH8A5ZSkfjrva1affiAC/iGeCo5Zh70zaMr2J0h0pgEb/r/l+gPZgh+zDoyRgbXkPY A+VzPS6cFSPjCE0b5wO/jwSHiSJSJpESP0fB8sEtux43mMG25ErxO9KV00eLFlvMwhXL/j NDbjMFKdR2/pPpLufspr7MZSuwxHDnDcSOp1C23uGwQ1YnGo/tS+Zt8obATbMU6QkAn/Hh 8HQ87SJmjEIeISdRpsRVCdcnjJ8YIw9QhyPDAXe5X0RRa/GBAw527kCbZIxCMfAl8++VZK uXT6OVIQp7HrS+jC3cM/RP2h4wK53gN2lsMIBHXa0Ee04fwzdezMGwk6T/uYkw== Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com [66.111.4.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Rc19k0Yl8zxsx; Thu, 31 Aug 2023 12:46:01 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id 9441527C005A; Thu, 31 Aug 2023 08:46:01 -0400 (EDT) Received: from imap51 ([10.202.2.101]) by compute5.internal (MEProxy); Thu, 31 Aug 2023 08:46:01 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudegtddgheeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerredtnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpeelveevkefhhfetudefffevtdfguefguefgtdekveekvefg fffffffhjefgieeuheenucffohhmrghinhepfhhrvggvsghsugdrohhrghdplhhisghnrg hmvghsrdhmkhdpshgvrhhvihgtvgdrihhnpdgtohhmphhilhgvrhdrmhhkpdhshihsrdhm khdplhhinhhkvghrrdhmkhdpohhpthhsrdhmkhdpohhsrdgsuhhilhgunecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghgrlhhlrghtihhnodhm vghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqddufeefheelvddvudeiqddvleehtd egudekgedqghgrlhhlrghtihhnpeepfhhrvggvsghsugdrohhrghesfhgrshhtmhgrihhl rdgtohhm X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 58C88B60089; Thu, 31 Aug 2023 08:46:01 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-701-g9b2f44d3ee-fm-20230823.001-g9b2f44d3 List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 Message-Id: <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> In-Reply-To: References: <202308270509.37R596B5048298@gitrepo.freebsd.org> Date: Thu, 31 Aug 2023 07:45:41 -0500 From: "Drew Gallatin" To: "Alexander Motin" , "Gleb Smirnoff" , "Martin Matuska" Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad Content-Type: multipart/alternative; boundary=06e1cace14e44ba6b615e2ddd8eabb7c --06e1cace14e44ba6b615e2ddd8eabb7c Content-Type: text/plain On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote: > Hi Gleb, > > It is the first time I see a panic like this. I'll think about it > tomorrow. But I'd appreciate any information on what is your workload > and what are you doing related to ZIL (O_SYNC, fsync(), sync=always, > etc) to trigger it? What is your pool configuration? I'm not Gleb, but this was something at $WORK, so I can perhaps help. I've included the output of zpool status, and all non-default settings in the zpool. Note that we don't use a ZIL device. Thanks, Drew # zpool status pool: zroot state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 nda0p2 ONLINE 0 0 0 nda1p2 ONLINE 0 0 0 nda2p2 ONLINE 0 0 0 spares nda3p2 AVAIL errors: No known data errors # zpool get all | grep -v default NAME PROPERTY VALUE SOURCE zroot size 149G - zroot capacity 14% - zroot health ONLINE - zroot guid 5099990586719219765 - zroot bootfs zroot/ROOT/NCD-7827-freebsd-2023-08-29-merge-oca-r202308291935 local zroot dedupratio 1.00x - zroot free 127G - zroot allocated 22.3G - zroot readonly off - zroot expandsize - - zroot freeing 0 - zroot fragmentation 28% - zroot leaked 0 - zroot checkpoint - - zroot load_guid 7612195721260396606 - zroot bcloneused 0 - zroot bclonesaved 0 - zroot bcloneratio 1.00x - zroot feature@async_destroy enabled local zroot feature@empty_bpobj active local zroot feature@lz4_compress active local zroot feature@multi_vdev_crash_dump enabled local zroot feature@spacemap_histogram active local zroot feature@enabled_txg active local zroot feature@hole_birth active local zroot feature@extensible_dataset active local zroot feature@embedded_data active local zroot feature@bookmarks enabled local zroot feature@filesystem_limits enabled local zroot feature@large_blocks enabled local zroot feature@large_dnode enabled local zroot feature@sha512 enabled local zroot feature@skein enabled local zroot feature@edonr disabled local zroot feature@userobj_accounting active local zroot feature@encryption enabled local zroot feature@project_quota active local zroot feature@device_removal enabled local zroot feature@obsolete_counts enabled local zroot feature@zpool_checkpoint enabled local zroot feature@spacemap_v2 active local zroot feature@allocation_classes enabled local zroot feature@resilver_defer enabled local zroot feature@bookmark_v2 enabled local zroot feature@redaction_bookmarks enabled local zroot feature@redacted_datasets enabled local zroot feature@bookmark_written enabled local zroot feature@log_spacemap active local zroot feature@livelist enabled local zroot feature@device_rebuild enabled local zroot feature@zstd_compress enabled local zroot feature@draid enabled local zroot feature@zilsaxattr disabled local zroot feature@head_errlog disabled local zroot feature@blake3 disabled local zroot feature@block_cloning disabled local zroot feature@vdev_zaps_v2 disabled local > On 30.08.2023 20:09, Gleb Smirnoff wrote: > > Hi Martin, > > > > we hit this panic with this update: > > > > panic: VERIFY(avl_is_empty(&lwb->lwb_vdev_tree)) failed > > > > cpuid = 8 > > time = 1693432004 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe09519aa800 > > vpanic() at vpanic+0x132/frame 0xfffffe09519aa930 > > spl_panic() at spl_panic+0x3a/frame 0xfffffe09519aa990 > > zil_free_lwb() at zil_free_lwb+0x1de/frame 0xfffffe09519aa9c0 > > zil_sync() at zil_sync+0x3cb/frame 0xfffffe09519aaaa0 > > [Wed Aug 30 21:46:44 2023]dmu_objset_sync() at dmu_objset_sync+0x51b/frame 0xfffffe09519aab70 > > dsl_pool_sync() at dsl_pool_sync+0x11d/frame 0xfffffe09519aabf0 > > spa_sync() at spa_sync+0xc68/frame 0xfffffe09519aae20 > > txg_sync_thread() at txg_sync_thread+0x2eb/frame 0xfffffe09519aaef0 > > fork_exit() at fork_exit+0x82/frame 0xfffffe09519aaf30 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe09519aaf30 > > --- trap 0x26d90a2f, rip = 0xbba54c81b25fd8d4, rsp = 0xff60c2f6f9874bcd, rbp = 0x37940e3e9793bd40 -- > > > > I have two cores dumped for this panic. > > > > Our previous version is based on 9c0a2d522f3a18fc795a7dd6266d2be2aff4b084, which > > I guess has the most recent ZFS merge openzfs/zfs@009d3288d. > > > > On Sun, Aug 27, 2023 at 05:09:06AM +0000, Martin Matuska wrote: > > M> The branch main has been updated by mm: > > M> > > M> URL: https://cgit.FreeBSD.org/src/commit/?id=315ee00fa9616b0a192b6834911f98bcf5316a6b > > M> > > M> commit 315ee00fa9616b0a192b6834911f98bcf5316a6b > > M> Merge: b3a714652ff0 804414aad224 > > M> Author: Martin Matuska > > M> AuthorDate: 2023-08-26 21:20:04 +0000 > > M> Commit: Martin Matuska > > M> CommitDate: 2023-08-26 21:51:42 +0000 > > M> > > M> zfs: merge openzfs/zfs@804414aad > > M> > > M> Notable upstream pull request merges: > > M> #15024 Add missed DMU_PROJECTUSED_OBJECT prefetch > > M> #15029 Do not request data L1 buffers on scan prefetch > > M> #15036 FreeBSD: catch up to __FreeBSD_version 1400093 > > M> #15039 Fix raw receive with different indirect block size > > M> #15047 FreeBSD: Fix build on stable/13 after 1302506 > > M> #15049 Fix the ZFS checksum error histograms with larger record sizes > > M> #15052 Reduce bloat in ereport.fs.zfs.checksum events > > M> #15056 Avoid extra snprintf() in dsl_deadlist_merge() > > M> #15061 Ignore pool ashift property during vdev attachment > > M> #15063 Don't panic if setting vdev properties is unsupported for this vdev type > > M> #15067 spa_min_alloc should be GCD, not min > > M> #15071 Add explicit prefetches to bpobj_iterate() > > M> #15072 Adjust prefetch parameters > > M> #15076 Refactor dmu_prefetch() > > M> #15079 set autotrim default to 'off' everywhere > > M> #15080 ZIL: Fix config lock deadlock > > M> #15088 metaslab: tuneable to better control force ganging > > M> #15096 Avoid waiting in dmu_sync_late_arrival() > > M> #15097 BRT should return EOPNOTSUPP > > M> #15103 Remove zl_issuer_lock from zil_suspend() > > M> #15107 Remove fastwrite mechanism > > M> #15113 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1 > > M> #15122 ZIL: Second attempt to reduce scope of zl_issuer_lock > > M> #15129 zpool_vdev_remove() should handle EALREADY error return > > M> #15132 ZIL: Replay blocks without next block pointer > > M> #15148 zfs_clone_range should return descriptive error codes > > M> #15153 ZIL: Avoid dbuf_read() before dmu_sync() > > M> #15172 copy_file_range: fix fallback when source create on same txg > > M> #15180 Update outdated assertion from zio_write_compress > > M> > > M> Obtained from: OpenZFS > > M> OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13 > > M> > > M> Makefile.inc1 | 5 +- > > M> cddl/lib/libzfs/Makefile | 1 + > > M> cddl/lib/libzfs/Makefile.depend | 1 + > > M> rescue/rescue/Makefile | 2 +- > > M> rescue/rescue/Makefile.depend | 1 + > > M> share/mk/src.libnames.mk | 2 +- > > M> sys/contrib/openzfs/META | 6 +- > > M> sys/contrib/openzfs/cmd/zdb/zdb.c | 77 ++- > > M> sys/contrib/openzfs/cmd/zed/agents/zfs_mod.c | 2 - > > M> sys/contrib/openzfs/cmd/zed/agents/zfs_retire.c | 5 + > > M> .../openzfs/cmd/zed/zed.d/statechange-slot_off.sh | 61 ++ > > M> sys/contrib/openzfs/cmd/zed/zed.d/zed.rc | 5 + > > M> sys/contrib/openzfs/cmd/ztest.c | 2 +- > > M> sys/contrib/openzfs/config/Shellcheck.am | 3 +- > > M> sys/contrib/openzfs/config/kernel-blkdev.m4 | 138 +++- > > M> .../config/kernel-block-device-operations.m4 | 35 +- > > M> .../openzfs/config/kernel-filemap-splice-read.m4 | 25 + > > M> .../openzfs/config/kernel-register_sysctl_table.m4 | 27 + > > M> .../config/kernel-vfs-extended-file_range.m4 | 50 ++ > > M> .../openzfs/config/kernel-vfs-file_range.m4 | 164 +++++ > > M> sys/contrib/openzfs/config/kernel-vfs-iov_iter.m4 | 26 +- > > M> sys/contrib/openzfs/config/kernel.m4 | 16 + > > M> sys/contrib/openzfs/contrib/debian/changelog | 6 + > > M> .../contrib/debian/openzfs-zfs-test.install | 2 - > > M> .../contrib/debian/openzfs-zfsutils.install | 2 + > > M> .../contrib/dracut/90zfs/zfs-env-bootfs.service.in | 11 +- > > M> .../dracut/90zfs/zfs-rollback-bootfs.service.in | 2 +- > > M> .../openzfs/include/os/freebsd/spl/sys/vnode.h | 6 +- > > M> .../include/os/linux/kernel/linux/blkdev_compat.h | 17 + > > M> .../openzfs/include/os/linux/spl/sys/kmem_cache.h | 8 + > > M> .../openzfs/include/os/linux/spl/sys/types.h | 2 +- > > M> sys/contrib/openzfs/include/os/linux/spl/sys/uio.h | 12 + > > M> sys/contrib/openzfs/include/os/linux/zfs/sys/zpl.h | 53 ++ > > M> sys/contrib/openzfs/include/sys/bpobj.h | 2 +- > > M> sys/contrib/openzfs/include/sys/brt.h | 1 + > > M> sys/contrib/openzfs/include/sys/dmu.h | 5 + > > M> sys/contrib/openzfs/include/sys/dmu_impl.h | 2 - > > M> sys/contrib/openzfs/include/sys/dmu_zfetch.h | 2 - > > M> sys/contrib/openzfs/include/sys/fm/fs/zfs.h | 4 - > > M> sys/contrib/openzfs/include/sys/metaslab.h | 3 - > > M> sys/contrib/openzfs/include/sys/metaslab_impl.h | 6 +- > > M> sys/contrib/openzfs/include/sys/spa_impl.h | 1 + > > M> sys/contrib/openzfs/include/sys/vdev_impl.h | 2 +- > > M> sys/contrib/openzfs/include/sys/zil_impl.h | 44 +- > > M> sys/contrib/openzfs/include/sys/zio.h | 1 - > > M> sys/contrib/openzfs/include/sys/zio_checksum.h | 2 - > > M> sys/contrib/openzfs/lib/libzfs/Makefile.am | 2 +- > > M> sys/contrib/openzfs/lib/libzfs/libzfs_pool.c | 6 + > > M> sys/contrib/openzfs/lib/libzfs/libzfs_sendrecv.c | 95 ++- > > M> sys/contrib/openzfs/man/man4/zfs.4 | 10 +- > > M> sys/contrib/openzfs/man/man8/zfs-send.8 | 18 +- > > M> sys/contrib/openzfs/man/man8/zpool-events.8 | 23 +- > > M> sys/contrib/openzfs/module/Kbuild.in | 1 + > > M> .../openzfs/module/os/freebsd/zfs/zfs_vnops_os.c | 10 +- > > M> sys/contrib/openzfs/module/os/linux/spl/spl-proc.c | 26 +- > > M> .../openzfs/module/os/linux/zfs/vdev_disk.c | 65 +- > > M> .../openzfs/module/os/linux/zfs/zfs_ctldir.c | 26 +- > > M> .../openzfs/module/os/linux/zfs/zfs_vfsops.c | 7 + > > M> .../openzfs/module/os/linux/zfs/zfs_vnops_os.c | 9 +- > > M> .../openzfs/module/os/linux/zfs/zfs_znode.c | 8 + > > M> .../openzfs/module/os/linux/zfs/zpl_ctldir.c | 2 +- > > M> sys/contrib/openzfs/module/os/linux/zfs/zpl_file.c | 44 +- > > M> .../openzfs/module/os/linux/zfs/zpl_file_range.c | 272 ++++++++ > > M> .../openzfs/module/os/linux/zfs/zpl_super.c | 39 +- > > M> sys/contrib/openzfs/module/os/linux/zfs/zvol_os.c | 28 +- > > M> sys/contrib/openzfs/module/zfs/bpobj.c | 49 +- > > M> sys/contrib/openzfs/module/zfs/brt.c | 43 +- > > M> sys/contrib/openzfs/module/zfs/dbuf.c | 11 +- > > M> sys/contrib/openzfs/module/zfs/dmu.c | 127 ++-- > > M> sys/contrib/openzfs/module/zfs/dmu_recv.c | 22 +- > > M> sys/contrib/openzfs/module/zfs/dmu_zfetch.c | 12 +- > > M> sys/contrib/openzfs/module/zfs/dnode.c | 31 +- > > M> sys/contrib/openzfs/module/zfs/dsl_deadlist.c | 14 +- > > M> sys/contrib/openzfs/module/zfs/dsl_scan.c | 17 +- > > M> sys/contrib/openzfs/module/zfs/metaslab.c | 83 +-- > > M> sys/contrib/openzfs/module/zfs/spa_log_spacemap.c | 4 +- > > M> sys/contrib/openzfs/module/zfs/spa_misc.c | 1 + > > M> sys/contrib/openzfs/module/zfs/vdev.c | 90 ++- > > M> sys/contrib/openzfs/module/zfs/vdev_indirect.c | 2 +- > > M> sys/contrib/openzfs/module/zfs/vdev_raidz.c | 2 +- > > M> sys/contrib/openzfs/module/zfs/zfs_fm.c | 33 +- > > M> sys/contrib/openzfs/module/zfs/zfs_vnops.c | 46 +- > > M> sys/contrib/openzfs/module/zfs/zil.c | 719 +++++++++------------ > > M> sys/contrib/openzfs/module/zfs/zio.c | 45 +- > > M> sys/contrib/openzfs/module/zfs/zio_checksum.c | 2 - > > M> sys/contrib/openzfs/module/zfs/zvol.c | 8 +- > > M> sys/contrib/openzfs/tests/runfiles/linux.run | 11 + > > M> .../openzfs/tests/test-runner/bin/zts-report.py.in | 18 + > > M> sys/contrib/openzfs/tests/zfs-tests/cmd/.gitignore | 1 + > > M> .../openzfs/tests/zfs-tests/cmd/Makefile.am | 1 + > > M> .../openzfs/tests/zfs-tests/cmd/clonefile.c | 333 ++++++++++ > > M> sys/contrib/openzfs/tests/zfs-tests/cmd/readmmap.c | 1 + > > M> .../openzfs/tests/zfs-tests/include/commands.cfg | 1 + > > M> .../openzfs/tests/zfs-tests/tests/Makefile.am | 14 + > > M> .../functional/block_cloning/block_cloning.kshlib | 54 ++ > > M> .../block_cloning/block_cloning_copyfilerange.ksh | 60 ++ > > M> .../block_cloning_copyfilerange_cross_dataset.ksh | 65 ++ > > M> .../block_cloning_copyfilerange_fallback.ksh | 86 +++ > > M> ...ock_cloning_copyfilerange_fallback_same_txg.ksh | 66 ++ > > M> .../block_cloning_copyfilerange_partial.ksh | 68 ++ > > M> .../block_cloning_disabled_copyfilerange.ksh | 60 ++ > > M> .../block_cloning_disabled_ficlone.ksh | 50 ++ > > M> .../block_cloning_disabled_ficlonerange.ksh | 50 ++ > > M> .../block_cloning/block_cloning_ficlone.ksh | 56 ++ > > M> .../block_cloning/block_cloning_ficlonerange.ksh | 56 ++ > > M> .../block_cloning_ficlonerange_partial.ksh | 64 ++ > > M> .../tests/functional/block_cloning/cleanup.ksh | 34 + > > M> .../tests/functional/block_cloning/setup.ksh | 36 ++ > > M> .../cli_root/zpool_attach/attach-o_ashift.ksh | 30 +- > > M> .../cli_root/zpool_replace/replace-o_ashift.ksh | 32 +- > > M> .../cli_root/zpool_replace/replace_prop_ashift.ksh | 24 +- > > M> sys/modules/zfs/zfs_config.h | 53 +- > > M> sys/modules/zfs/zfs_gitrev.h | 2 +- > > M> 113 files changed, 3322 insertions(+), 906 deletions(-) > > M> > > M> diff --cc Makefile.inc1 > > M> index 8838d31c795a,000000000000..f740cc5abcca > > M> mode 100644,000000..100644 > > M> --- a/Makefile.inc1 > > M> +++ b/Makefile.inc1 > > M> @@@ -1,3692 -1,0 +1,3695 @@@ > > M> +# > > M> +# > > M> +# Make command line options: > > M> +# -DNO_CLEANDIR run ${MAKE} clean, instead of ${MAKE} cleandir > > M> +# -DNO_CLEAN do not clean at all > > M> +# -DDB_FROM_SRC use the user/group databases in src/etc instead of > > M> +# the system database when installing. > > M> +# -DNO_SHARE do not go into share subdir > > M> +# -DKERNFAST define NO_KERNEL{CONFIG,CLEAN,OBJ} > > M> +# -DNO_KERNELCONFIG do not run config in ${MAKE} buildkernel > > M> +# -DNO_KERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel > > M> +# -DNO_KERNELOBJ do not run ${MAKE} obj in ${MAKE} buildkernel > > M> +# -DNO_ROOT install without using root privilege > > M> +# -DWITHOUT_CTF do not run the DTrace CTF conversion tools on built objects > > M> +# LOCAL_DIRS="list of dirs" to add additional dirs to the SUBDIR list > > M> +# LOCAL_ITOOLS="list of tools" to add additional tools to the ITOOLS list > > M> +# LOCAL_LIB_DIRS="list of dirs" to add additional dirs to libraries target > > M> +# LOCAL_MTREE="list of mtree files" to process to allow local directories > > M> +# to be created before files are installed > > M> +# LOCAL_LEGACY_DIRS="list of dirs" to add additional dirs to the legacy > > M> +# target > > M> +# LOCAL_BSTOOL_DIRS="list of dirs" to add additional dirs to the > > M> +# bootstrap-tools target > > M> +# LOCAL_TOOL_DIRS="list of dirs" to add additional dirs to the build-tools > > M> +# target > > M> +# LOCAL_XTOOL_DIRS="list of dirs" to add additional dirs to the > > M> +# cross-tools target > > M> +# METALOG="path to metadata log" to write permission and ownership > > M> +# when NO_ROOT is set. (default: ${DESTDIR}/${DISTDIR}/METALOG, > > M> +# check /etc/make.conf for DISTDIR) > > M> +# TARGET="machine" to crossbuild world for a different machine type > > M> +# TARGET_ARCH= may be required when a TARGET supports multiple endians > > M> +# BUILDENV_SHELL= shell to launch for the buildenv target (def:${SHELL}) > > M> +# WORLD_FLAGS= additional flags to pass to make(1) during buildworld > > M> +# KERNEL_FLAGS= additional flags to pass to make(1) during buildkernel > > M> +# SUBDIR_OVERRIDE="list of dirs" to build rather than everything. > > M> +# All libraries and includes, and some build tools will still build. > > M> + > > M> +# > > M> +# The intended user-driven targets are: > > M> +# buildworld - rebuild *everything*, including glue to help do upgrades > > M> +# installworld- install everything built by "buildworld" > > M> +# checkworld - run test suite on installed world > > M> +# doxygen - build API documentation of the kernel > > M> +# > > M> +# Standard targets (not defined here) are documented in the makefiles in > > M> +# /usr/share/mk. These include: > > M> +# obj depend all install clean cleandepend cleanobj > > M> + > > M> +.if !defined(TARGET) || !defined(TARGET_ARCH) > > M> +.error Both TARGET and TARGET_ARCH must be defined. > > M> +.endif > > M> + > > M> +.if make(showconfig) || make(test-system-*) > > M> +_MKSHOWCONFIG= t > > M> +.endif > > M> + > > M> +SRCDIR?= ${.CURDIR} > > M> +LOCALBASE?= /usr/local > > M> +TIME_ENV ?= time env > > M> + > > M> +.include "share/mk/src.tools.mk" > > M> + > > M> +# Cross toolchain changes must be in effect before bsd.compiler.mk > > M> +# so that gets the right CC, and pass CROSS_TOOLCHAIN to submakes. > > M> +.if defined(CROSS_TOOLCHAIN) > > M> +.if exists(${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk) > > M> +.include "${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk" > > M> +.elif exists(${CROSS_TOOLCHAIN}) > > M> +.include "${CROSS_TOOLCHAIN}" > > M> +.else > > M> +.error CROSS_TOOLCHAIN ${CROSS_TOOLCHAIN} not found > > M> +.endif > > M> +CROSSENV+=CROSS_TOOLCHAIN="${CROSS_TOOLCHAIN}" > > M> +.elif defined(UNIVERSE_TOOLCHAIN) > > M> +UNIVERSE_TOOLCHAIN_PATH?=${HOST_OBJTOP}/tmp/usr/bin > > M> +XCC?="${UNIVERSE_TOOLCHAIN_PATH}/cc" > > M> +XCXX?="${UNIVERSE_TOOLCHAIN_PATH}/c++" > > M> +XCPP?="${UNIVERSE_TOOLCHAIN_PATH}/cpp" > > M> +XLD?="${UNIVERSE_TOOLCHAIN_PATH}/ld" > > M> +.endif > > M> +.if defined(CROSS_TOOLCHAIN_PREFIX) > > M> +CROSS_COMPILER_PREFIX?=${CROSS_TOOLCHAIN_PREFIX} > > M> +.endif > > M> + > > M> +XCOMPILERS= CC CXX CPP > > M> +.for COMPILER in ${XCOMPILERS} > > M> +.if defined(CROSS_COMPILER_PREFIX) > > M> +X${COMPILER}?= ${CROSS_COMPILER_PREFIX}${${COMPILER}} > > M> +.else > > M> +X${COMPILER}?= ${${COMPILER}} > > M> +.endif > > M> +.endfor > > M> +# If a full path to an external cross compiler is given, don't build > > M> +# a cross compiler. > > M> +.if ${XCC:N${CCACHE_BIN}:M/*} > > M> +MK_CLANG_BOOTSTRAP= no > > M> +# Make sure sub-makes see the option as disabled so the hack in bsd.sys.mk to > > M> +# work around incompatible headers in Clang's resource directory is enabled. > > M> +.MAKEOVERRIDES+= MK_CLANG_BOOTSTRAP > > M> +.endif > > M> + > > M> +# Pull in compiler metadata from buildworld/toolchain if possible to avoid > > M> +# running CC from bsd.compiler.mk. > > M> +.if make(installworld) || make(install) || make(distributeworld) || \ > > M> + make(stageworld) > > M> +.-include "${OBJTOP}/toolchain-metadata.mk" > > M> +.if !defined(_LOADED_TOOLCHAIN_METADATA) > > M> +.error A build is required first. You may have the wrong MAKEOBJDIRPREFIX set. > > M> +.endif > > M> +.endif > > M> + > > M> +# Pull in COMPILER_TYPE and COMPILER_FREEBSD_VERSION early. Pull it from the > > M> +# tree to be friendlier to foreign OS builds. It's safe to do so unconditionally > > M> +# here since we will always have the right make, unlike in src/Makefile > > M> +# Don't include bsd.linker.mk yet until XBINUTILS is handled (after src.opts.mk) > > M> +_NO_INCLUDE_LINKERMK= t > > M> +# We also want the X_COMPILER* variables if we are using an external toolchain. > > M> +_WANT_TOOLCHAIN_CROSS_VARS= t > > M> +.include "share/mk/bsd.compiler.mk" > > M> +.undef _NO_INCLUDE_LINKERMK > > M> +.undef _WANT_TOOLCHAIN_CROSS_VARS > > M> +# src.opts.mk depends on COMPILER_FEATURES > > M> +.include "share/mk/src.opts.mk" > > M> + > > M> +.if ${TARGET} == ${MACHINE} > > M> +TARGET_CPUTYPE?=${CPUTYPE} > > M> +.else > > M> +TARGET_CPUTYPE?= > > M> +.endif > > M> +.if !empty(TARGET_CPUTYPE) > > M> +_TARGET_CPUTYPE=${TARGET_CPUTYPE} > > M> +.else > > M> +_TARGET_CPUTYPE=dummy > > M> +.endif > > M> +.if ${TARGET} == "arm" > > M> +.if ${TARGET_CPUTYPE:M*soft*} == "" > > M> +TARGET_TRIPLE_ABI= gnueabihf > > M> +.else > > M> +TARGET_TRIPLE_ABI= gnueabi > > M> +.endif > > M> +.endif > > M> +MACHINE_TRIPLE_ABI?= unknown > > M> +MACHINE_TRIPLE?=${MACHINE_ARCH:S/amd64/x86_64/}-${MACHINE_TRIPLE_ABI}-freebsd${OS_REVISION} > > M> +TARGET_TRIPLE_ABI?= unknown > > M> +TARGET_TRIPLE?= ${TARGET_ARCH:S/amd64/x86_64/}-${TARGET_TRIPLE_ABI}-freebsd${OS_REVISION} > > M> +KNOWN_ARCHES?= aarch64/arm64 \ > > M> + amd64 \ > > M> + armv6/arm \ > > M> + armv7/arm \ > > M> + i386 \ > > M> + powerpc \ > > M> + powerpc64/powerpc \ > > M> + powerpc64le/powerpc \ > > M> + powerpcspe/powerpc \ > > M> + riscv64/riscv > > M> + > > M> +.if ${TARGET} == ${TARGET_ARCH} > > M> +_t= ${TARGET} > > M> +.else > > M> +_t= ${TARGET_ARCH}/${TARGET} > > M> +.endif > > M> +.for _t in ${_t} > > M> +.if empty(KNOWN_ARCHES:M${_t}) > > M> +.error Unknown target ${TARGET_ARCH}:${TARGET}. > > M> +.endif > > M> +.endfor > > M> + > > M> +.if ${TARGET_ARCH} == "amd64" > > M> +LIBCOMPAT_INCLUDE_DIRS+= i386 > > M> +.elif ${TARGET_ARCH} == "aarch64" > > M> +LIBCOMPAT_INCLUDE_DIRS+= arm > > M> +.endif > > M> + > > M> +.if ${.MAKE.OS} != "FreeBSD" > > M> +CROSSBUILD_HOST=${.MAKE.OS} > > M> +.if ${.MAKE.OS} != "Linux" && ${.MAKE.OS} != "Darwin" > > M> +.warning Unsupported crossbuild system: ${.MAKE.OS}. Build will probably fail! > > M> +.endif > > M> +# We need to force NO_ROOT/DB_FROM_SRC builds when building on other operating > > M> +# systems since the BSD.foo.dist specs contain users and groups that do not > > M> +# exist by default on a Linux/MacOS system. > > M> +NO_ROOT:= 1 > > M> +DB_FROM_SRC:= 1 > > M> +.export NO_ROOT > > M> +.endif > > M> + > > M> +# If all targets are disabled for system llvm then don't expect it to work > > M> +# for cross-builds. > > M> +.if !defined(TOOLS_PREFIX) && ${MK_LLVM_TARGET_ALL} == "no" && \ > > M> + ${MACHINE} != ${TARGET} && ${MACHINE_ARCH} != ${TARGET_ARCH} && \ > > M> + !make(showconfig) > > M> +MK_SYSTEM_COMPILER= no > > M> +MK_SYSTEM_LINKER= no > > M> +.endif > > M> + > > M> +# Handle external binutils. > > M> +.if defined(CROSS_TOOLCHAIN_PREFIX) > > M> +CROSS_BINUTILS_PREFIX?=${CROSS_TOOLCHAIN_PREFIX} > > M> +.endif > > M> +XBINUTILS= AS AR ELFCTL LD NM OBJCOPY RANLIB SIZE STRINGS STRIPBIN > > M> +.for BINUTIL in ${XBINUTILS} > > M> +.if defined(CROSS_BINUTILS_PREFIX) && \ > > M> + exists(${CROSS_BINUTILS_PREFIX}/${${BINUTIL}}) > > M> +X${BINUTIL}?= ${CROSS_BINUTILS_PREFIX:C,/*$,,}/${${BINUTIL}} > > M> +.else > > M> +X${BINUTIL}?= ${${BINUTIL}} > > M> +.endif > > M> +.endfor > > M> + > > M> +# If a full path to an external linker is given, don't build lld. > > M> +.if ${XLD:M/*} > > M> +MK_LLD_BOOTSTRAP= no > > M> +.endif > > M> + > > M> +# We also want the X_LINKER* variables if we are using an external toolchain. > > M> +_WANT_TOOLCHAIN_CROSS_VARS= t > > M> +.include "share/mk/bsd.linker.mk" > > M> +.undef _WANT_TOOLCHAIN_CROSS_VARS > > M> + > > M> +# Begin WITH_SYSTEM_COMPILER / WITH_SYSTEM_LD > > M> + > > M> +# WITH_SYSTEM_COMPILER - Pull in needed values and make a decision. > > M> + > > M> +# Check if there is a local compiler that can satisfy as an external compiler. > > M> +# Which compiler is expected to be used? > > M> +.if ${MK_CLANG_BOOTSTRAP} == "yes" > > M> +WANT_COMPILER_TYPE= clang > > M> +.else > > M> +WANT_COMPILER_TYPE= > > M> +.endif > > M> + > > M> +.if !defined(WANT_COMPILER_FREEBSD_VERSION) && !make(showconfig) && \ > > M> + !make(test-system-linker) > > M> +.if ${WANT_COMPILER_TYPE} == "clang" > > M> +WANT_COMPILER_FREEBSD_VERSION_FILE= lib/clang/freebsd_cc_version.h > > M> +WANT_COMPILER_FREEBSD_VERSION!= \ > > M> + awk '$$2 == "FREEBSD_CC_VERSION" {printf("%d\n", $$3)}' \ > > M> + ${SRCDIR}/${WANT_COMPILER_FREEBSD_VERSION_FILE} || echo unknown > > M> +WANT_COMPILER_VERSION_FILE= lib/clang/include/clang/Basic/Version.inc > > M> +WANT_COMPILER_VERSION!= \ > > M> + awk '$$2 == "CLANG_VERSION" {split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \ > > M> + ${SRCDIR}/${WANT_COMPILER_VERSION_FILE} || echo unknown > > M> +.endif > > M> +.export WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_VERSION > > M> +.endif # !defined(WANT_COMPILER_FREEBSD_VERSION) > > M> + > > M> +# It needs to be the same revision as we would build for the bootstrap. > > M> +# If the expected vs CC is different then we can't skip. > > M> +# GCC cannot be used for cross-arch yet. For clang we pass -target later if > > M> +# TARGET_ARCH!=MACHINE_ARCH. > > M> +.if ${MK_SYSTEM_COMPILER} == "yes" && \ > > M> + defined(WANT_COMPILER_FREEBSD_VERSION) && \ > > M> + ${MK_CLANG_BOOTSTRAP} == "yes" && \ > > M> + !make(xdev*) && \ > > M> + ${X_COMPILER_TYPE} == ${WANT_COMPILER_TYPE} && \ > > M> + (${X_COMPILER_TYPE} == "clang" || ${TARGET_ARCH} == ${MACHINE_ARCH}) && \ > > M> + ${X_COMPILER_VERSION} == ${WANT_COMPILER_VERSION} && \ > > M> + ${X_COMPILER_FREEBSD_VERSION} == ${WANT_COMPILER_FREEBSD_VERSION} > > M> +# Everything matches, disable the bootstrap compiler. > > M> +MK_CLANG_BOOTSTRAP= no > > M> +USING_SYSTEM_COMPILER= yes > > M> +.endif # ${WANT_COMPILER_TYPE} == ${COMPILER_TYPE} > > M> + > > M> +# WITH_SYSTEM_LD - Pull in needed values and make a decision. > > M> + > > M> +# Check if there is a local linker that can satisfy as an external linker. > > M> +# Which linker is expected to be used? > > M> +.if ${MK_LLD_BOOTSTRAP} == "yes" > > M> +WANT_LINKER_TYPE= lld > > M> +.else > > M> +WANT_LINKER_TYPE= > > M> +.endif > > M> + > > M> +.if !defined(WANT_LINKER_FREEBSD_VERSION) && !make(showconfig) && \ > > M> + !make(test-system-compiler) > > M> +.if ${WANT_LINKER_TYPE} == "lld" > > M> +WANT_LINKER_FREEBSD_VERSION_FILE= lib/clang/include/lld/Common/Version.inc > > M> +WANT_LINKER_FREEBSD_VERSION!= \ > > M> + awk '$$2 == "LLD_FREEBSD_VERSION" {print $$3}' \ > > M> + ${SRCDIR}/${WANT_LINKER_FREEBSD_VERSION_FILE} || echo unknown > > M> +WANT_LINKER_VERSION_FILE= lib/clang/include/lld/Common/Version.inc > > M> +WANT_LINKER_VERSION!= \ > > M> + awk '$$2 == "LLD_VERSION_STRING" {gsub("\"", "", $$3); split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \ > > M> + ${SRCDIR}/${WANT_LINKER_VERSION_FILE} || echo unknown > > M> +.else > > M> +WANT_LINKER_FREEBSD_VERSION_FILE= > > M> +WANT_LINKER_FREEBSD_VERSION= > > M> +.endif > > M> +.export WANT_LINKER_FREEBSD_VERSION WANT_LINKER_VERSION > > M> +.endif # !defined(WANT_LINKER_FREEBSD_VERSION) > > M> + > > M> +.if ${MK_SYSTEM_LINKER} == "yes" && \ > > M> + defined(WANT_LINKER_FREEBSD_VERSION) && \ > > M> + (${MK_LLD_BOOTSTRAP} == "yes") && \ > > M> + !make(xdev*) && \ > > M> + ${X_LINKER_TYPE} == ${WANT_LINKER_TYPE} && \ > > M> + ${X_LINKER_VERSION} == ${WANT_LINKER_VERSION} && \ > > M> + ${X_LINKER_FREEBSD_VERSION} == ${WANT_LINKER_FREEBSD_VERSION} > > M> +# Everything matches, disable the bootstrap linker. > > M> +MK_LLD_BOOTSTRAP= no > > M> +USING_SYSTEM_LINKER= yes > > M> +.endif # ${WANT_LINKER_TYPE} == ${LINKER_TYPE} > > M> + > > M> +# WITH_SYSTEM_COMPILER / WITH_SYSTEM_LINKER - Handle defaults and debug. > > M> +USING_SYSTEM_COMPILER?= no > > M> +USING_SYSTEM_LINKER?= no > > M> + > > M> +TEST_SYSTEM_COMPILER_VARS= \ > > M> + USING_SYSTEM_COMPILER MK_SYSTEM_COMPILER \ > > M> + MK_CROSS_COMPILER MK_CLANG_BOOTSTRAP \ > > M> + WANT_COMPILER_TYPE WANT_COMPILER_VERSION WANT_COMPILER_VERSION_FILE \ > > M> + WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_FREEBSD_VERSION_FILE \ > > M> + CC COMPILER_TYPE COMPILER_FEATURES COMPILER_VERSION \ > > M> + COMPILER_FREEBSD_VERSION \ > > M> + XCC X_COMPILER_TYPE X_COMPILER_FEATURES X_COMPILER_VERSION \ > > M> + X_COMPILER_FREEBSD_VERSION > > M> +TEST_SYSTEM_LINKER_VARS= \ > > M> + USING_SYSTEM_LINKER MK_SYSTEM_LINKER \ > > M> + MK_LLD_BOOTSTRAP \ > > M> + WANT_LINKER_TYPE WANT_LINKER_VERSION WANT_LINKER_VERSION_FILE \ > > M> + WANT_LINKER_FREEBSD_VERSION WANT_LINKER_FREEBSD_VERSION_FILE \ > > M> + LD LINKER_TYPE LINKER_FEATURES LINKER_VERSION \ > > M> + LINKER_FREEBSD_VERSION \ > > M> + XLD X_LINKER_TYPE X_LINKER_FEATURES X_LINKER_VERSION \ > > M> + X_LINKER_FREEBSD_VERSION > > M> + > > M> +.for _t in compiler linker > > M> +test-system-${_t}: .PHONY > > M> +.for v in ${TEST_SYSTEM_${_t:tu}_VARS} > > M> + ${_+_}@printf "%-35s= %s\n" "${v}" "${${v}}" > > M> +.endfor > > M> +.endfor > > M> +.if (make(buildworld) || make(buildkernel) || make(kernel-toolchain) || \ > > M> + make(toolchain) || make(_cross-tools)) > > M> +.if ${USING_SYSTEM_COMPILER} == "yes" > > M> +.info SYSTEM_COMPILER: Determined that CC=${CC} matches the source tree. Not bootstrapping a cross-compiler. > > M> +.elif ${MK_CLANG_BOOTSTRAP} == "yes" > > M> +.info SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler. > > M> +.endif > > M> +.if ${USING_SYSTEM_LINKER} == "yes" > > M> +.info SYSTEM_LINKER: Determined that LD=${LD} matches the source tree. Not bootstrapping a cross-linker. > > M> +.elif ${MK_LLD_BOOTSTRAP} == "yes" > > M> +.info SYSTEM_LINKER: libclang will be built for bootstrapping a cross-linker. > > M> +.endif > > M> +.endif > > M> + > > M> +# End WITH_SYSTEM_COMPILER / WITH_SYSTEM_LD > > M> + > > M> +# Store some compiler metadata for use in installworld where we don't > > M> +# want to invoke CC at all. > > M> +_TOOLCHAIN_METADATA_VARS= COMPILER_VERSION \ > > M> + COMPILER_TYPE \ > > M> + COMPILER_FEATURES \ > > M> + COMPILER_FREEBSD_VERSION \ > > M> + COMPILER_RESOURCE_DIR \ > > M> + LINKER_VERSION \ > > M> + LINKER_FEATURES \ > > M> + LINKER_TYPE \ > > M> + LINKER_FREEBSD_VERSION > > M> +toolchain-metadata.mk: .PHONY .META > > M> + @: > ${.TARGET} > > M> + @echo ".info Using cached toolchain metadata from build at $$(hostname) on $$(date)" \ > > M> + > ${.TARGET} > > M> + @echo "_LOADED_TOOLCHAIN_METADATA=t" >> ${.TARGET} > > M> +.for v in ${_TOOLCHAIN_METADATA_VARS} > > M> + @echo "${v}=${${v}}" >> ${.TARGET} > > M> + @echo "X_${v}=${X_${v}}" >> ${.TARGET} > > M> +.endfor > > M> + @echo ".export ${_TOOLCHAIN_METADATA_VARS}" >> ${.TARGET} > > M> + @echo ".export ${_TOOLCHAIN_METADATA_VARS:C,^,X_,}" >> ${.TARGET} > > M> + > > M> + > > M> +# We must do lib/ and libexec/ before bin/ in case of a mid-install error to > > M> +# keep the users system reasonably usable. For static->dynamic root upgrades, > > M> +# we don't want to install a dynamic binary without rtld and the needed > > M> +# libraries. More commonly, for dynamic root, we don't want to install a > > M> +# binary that requires a newer library version that hasn't been installed yet. > > M> +# This ordering is not a guarantee though. The only guarantee of a working > > M> +# system here would require fine-grained ordering of all components based > > M> +# on their dependencies. > > M> +.if !empty(SUBDIR_OVERRIDE) > > M> +SUBDIR= ${SUBDIR_OVERRIDE} > > M> +.else > > M> +SUBDIR= lib libexec > > M> +# Add LOCAL_LIB_DIRS, but only if they will not be picked up as a SUBDIR > > M> +# of a LOCAL_DIRS directory. This allows LOCAL_DIRS=foo and > > M> +# LOCAL_LIB_DIRS=foo/lib to behave as expected. > > M> +.for _DIR in ${LOCAL_DIRS:M*/} ${LOCAL_DIRS:N*/:S|$|/|} > > M> +_REDUNDANT_LIB_DIRS+= ${LOCAL_LIB_DIRS:M${_DIR}*} > > M> +.endfor > > M> +.for _DIR in ${LOCAL_LIB_DIRS} > > M> +.if ${_DIR} == ".WAIT" || (empty(_REDUNDANT_LIB_DIRS:M${_DIR}) && exists(${.CURDIR}/${_DIR}/Makefile)) > > M> +SUBDIR+= ${_DIR} > > M> +.endif > > M> +.endfor > > M> +.if !defined(NO_ROOT) && (make(installworld) || make(install)) > > M> +# Ensure libraries are installed before progressing. > > M> +SUBDIR+=.WAIT > > M> +.endif > > M> +SUBDIR+=bin > > M> +.if ${MK_CDDL} != "no" > > M> +SUBDIR+=cddl > > M> +.endif > > M> +SUBDIR+=gnu include > > M> +.if ${MK_KERBEROS} != "no" > > M> +SUBDIR+=kerberos5 > > M> +.endif > > M> +.if ${MK_RESCUE} != "no" > > M> +SUBDIR+=rescue > > M> +.endif > > M> +SUBDIR+=sbin > > M> +.if ${MK_CRYPT} != "no" > > M> +SUBDIR+=secure > > M> +.endif > > M> +.if !defined(NO_SHARE) > > M> +SUBDIR+=share > > M> +.endif > > M> +.if ${MK_BOOT} != "no" > > M> +SUBDIR+=stand > > M> +.endif > > M> +SUBDIR+=sys usr.bin usr.sbin > > M> +.if ${MK_TESTS} != "no" > > M> +SUBDIR+= tests > > M> +.endif > > M> + > > M> +# Local directories are built in parallel with the base system directories. > > M> +# Users may insert a .WAIT directive at the beginning or elsewhere within > > M> +# the LOCAL_DIRS and LOCAL_LIB_DIRS lists as needed. > > M> +.for _DIR in ${LOCAL_DIRS} > > M> +.if ${_DIR} == ".WAIT" || exists(${.CURDIR}/${_DIR}/Makefile) > > M> +SUBDIR+= ${_DIR} > > M> +.endif > > M> +.endfor > > M> + > > M> +# We must do etc/ last as it hooks into building the man whatis file > > M> +# by calling 'makedb' in share/man. This is only relevant for > > M> +# install/distribute so they build the whatis file after every manpage is > > M> +# installed. > > M> +.if make(installworld) || make(install) > > M> +SUBDIR+=.WAIT > > M> +.endif > > M> +SUBDIR+=etc > > M> + > > M> +.endif # !empty(SUBDIR_OVERRIDE) > > M> + > > M> +.if defined(NOCLEAN) > > M> +.warning The src.conf WITHOUT_CLEAN option can now be used instead of NOCLEAN. > > M> +MK_CLEAN:= no > > M> +.endif > > M> +.if defined(NO_CLEAN) > > M> +.info The src.conf WITHOUT_CLEAN option can now be used instead of NO_CLEAN. > > M> +MK_CLEAN:= no > > M> +.endif > > M> +.if defined(NO_CLEANDIR) > > M> +CLEANDIR= clean cleandepend > > M> +.else > > M> +CLEANDIR= cleandir > > M> +.endif > > M> + > > M> +.if defined(WORLDFAST) > > M> +MK_CLEAN:= no > > M> +NO_OBJWALK= t > > M> +.endif > > M> + > > M> +.if ${MK_META_MODE} == "yes" > > M> +# If filemon is used then we can rely on the build being incremental-safe. > > M> +# The .meta files will also track the build command and rebuild should > > M> +# it change. > > M> +.if empty(.MAKE.MODE:Mnofilemon) > > M> +MK_CLEAN:= no > > M> +.endif > > M> +.endif > > M> +.if defined(NO_OBJWALK) || ${MK_AUTO_OBJ} == "yes" > > M> +NO_OBJWALK= t > > M> +NO_KERNELOBJ= t > > M> +.endif > > M> +.if !defined(NO_OBJWALK) > > M> +_obj= obj > > M> +.endif > > M> + > > M> +LOCAL_TOOL_DIRS?= > > M> +PACKAGEDIR?= ${DESTDIR}/${DISTDIR} > > M> + > > M> +.if empty(SHELL:M*csh*) > > M> +BUILDENV_SHELL?=${SHELL} > > M> +.else > > M> +BUILDENV_SHELL?=/bin/sh > > M> +.endif > > M> + > > M> +.if !defined(_MKSHOWCONFIG) > > M> +.if !defined(VCS_REVISION) || empty(VCS_REVISION) > > M> +.if !defined(SVNVERSION_CMD) || empty(SVNVERSION_CMD) > > M> +. for _D in ${PATH:S,:, ,g} > > M> +. if exists(${_D}/svnversion) > > M> +SVNVERSION_CMD?=${_D}/svnversion > > M> +. endif > > M> +. if exists(${_D}/svnliteversion) > > M> +SVNVERSION_CMD?=${_D}/svnliteversion > > M> +. endif > > M> +. endfor > > M> +.endif > > M> +.if defined(SVNVERSION_CMD) && !empty(SVNVERSION_CMD) > > M> +_VCS_REVISION?= $$(eval ${SVNVERSION_CMD} ${SRCDIR}) > > M> +. if !empty(_VCS_REVISION) > > M> +VCS_REVISION= $$(echo r${_VCS_REVISION}) > > M> +.export VCS_REVISION > > M> +. endif > > M> +.endif > > M> +.endif > > M> + > > M> +.if !defined(GIT_CMD) || empty(GIT_CMD) > > M> +. for _P in /usr/bin /usr/local/bin > > M> +. if exists(${_P}/git) > > M> +GIT_CMD= ${_P}/git > > M> +. endif > > M> +. endfor > > M> +.export GIT_CMD > > M> +.endif > > M> + > > M> +.if !defined(OSRELDATE) > > M> +.if exists(/usr/include/osreldate.h) > > M> +OSRELDATE!= awk '/^\#define[[:space:]]*__FreeBSD_version/ { print $$3 }' \ > > M> + /usr/include/osreldate.h > > M> +.else > > M> +OSRELDATE= 0 > > M> +.endif > > M> +.export OSRELDATE > > M> +.endif > > M> + > > M> +# Set VERSION for CTFMERGE to use via the default CTFFLAGS=-L VERSION. > > M> +.for _V in BRANCH REVISION TYPE > > M> +.if !defined(_${_V}) > > M> +_${_V}!= eval $$(awk '/^${_V}=/{print}' ${SRCTOP}/sys/conf/newvers.sh); echo $$${_V} > > M> +.export _${_V} > > M> +.endif > > M> +.endfor > > M> +.if !defined(SRCRELDATE) > > M> +SRCRELDATE!= awk '/^\#define[[:space:]]*__FreeBSD_version/ { print $$3 }' \ > > M> + ${SRCDIR}/sys/sys/param.h > > M> +.export SRCRELDATE > > M> +.endif > > M> +.if !defined(VERSION) > > M> +VERSION= FreeBSD ${_REVISION}-${_BRANCH:C/-p[0-9]+$//} ${TARGET_ARCH} ${SRCRELDATE} > > M> +.export VERSION > > M> +.endif > > M> +MAJOR_REVISION= ${_REVISION:R} > > M> + > > M> +.if !defined(PKG_VERSION) > > M> +_STRTIMENOW= %Y%m%d%H%M%S > > M> +_TIMENOW= ${_STRTIMENOW:gmtime} > > M> +.if ${_BRANCH:MCURRENT*} || ${_BRANCH:MSTABLE*} || ${_BRANCH:MPRERELEASE*} > > M> +EXTRA_REVISION= .snap${_TIMENOW} > > M> +.elif ${_BRANCH:MALPHA*} > > M> +EXTRA_REVISION= .a${_BRANCH:C/ALPHA([0-9]+).*/\1/}.${_TIMENOW} > > M> +.elif ${_BRANCH:MBETA*} > > M> +EXTRA_REVISION= .b${_BRANCH:C/BETA([0-9]+).*/\1/}.${_TIMENOW} > > M> +.elif ${_BRANCH:MRC*} > > M> +EXTRA_REVISION= .rc${_BRANCH:C/RC([0-9]+).*/\1/}.${_TIMENOW} > > M> +.elif ${_BRANCH:M*-p*} > > M> +EXTRA_REVISION= p${_BRANCH:C/.*-p([0-9]+$)/\1/} > > M> +.endif > > M> +PKG_VERSION:= ${MAJOR_REVISION}${EXTRA_REVISION:C/[[:space:]]//g} > > M> +.endif > > M> +.endif # !defined(PKG_VERSION) > > M> + > > M> +.if !defined(PKG_TIMESTAMP) > > M> +TIMEEPOCHNOW= %s > > M> +SOURCE_DATE_EPOCH= ${TIMEEPOCHNOW:gmtime} > > M> +.else > > M> +SOURCE_DATE_EPOCH= ${PKG_TIMESTAMP} > > M> +.endif > > M> + > > M> +PKG_NAME_PREFIX?= FreeBSD > > M> +PKG_MAINTAINER?= re@FreeBSD.org > > M> +PKG_WWW?= https://www.FreeBSD.org > > M> +.export PKG_NAME_PREFIX > > M> +.export PKG_MAINTAINER > > M> +.export PKG_WWW > > M> + > > M> +.if !defined(_MKSHOWCONFIG) > > M> +_CPUTYPE!= MAKEFLAGS= CPUTYPE=${_TARGET_CPUTYPE} ${MAKE} -f /dev/null \ > > M> + -m ${.CURDIR}/share/mk MK_AUTO_OBJ=no -V CPUTYPE > > M> +.if ${_CPUTYPE} != ${_TARGET_CPUTYPE} > > M> +.error CPUTYPE global should be set with ?=. > > M> +.endif > > M> +.endif > > M> +.if make(buildworld) > > M> +BUILD_ARCH!= uname -p > > M> +# On some Linux systems uname -p returns "unknown" so skip this check there. > > M> +# This check only exists to tell people to use TARGET_ARCH instead of > > M> +# MACHINE_ARCH so skipping it when crossbuilding on non-FreeBSD should be fine. > > M> +.if ${MACHINE_ARCH} != ${BUILD_ARCH} && ${.MAKE.OS} == "FreeBSD" > > M> +.error To cross-build, set TARGET_ARCH. > > M> +.endif > > M> +.endif > > M> +WORLDTMP?= ${OBJTOP}/tmp > > M> +BPATH= ${CCACHE_WRAPPER_PATH_PFX}${WORLDTMP}/legacy/usr/sbin:${WORLDTMP}/legacy/usr/bin:${WORLDTMP}/legacy/bin:${WORLDTMP}/legacy/usr/libexec > > M> +XPATH= ${WORLDTMP}/bin:${WORLDTMP}/usr/sbin:${WORLDTMP}/usr/bin > > M> + > > M> +# When building we want to find the cross tools before the host tools in ${BPATH}. > > M> +# We also need to add UNIVERSE_TOOLCHAIN_PATH so that we can find the shared > > M> +# toolchain files (clang, lld, etc.) during make universe/tinderbox > > M> +STRICTTMPPATH= ${XPATH}:${BPATH}:${UNIVERSE_TOOLCHAIN_PATH} > > M> +# We should not be using tools from /usr/bin accidentally since this could cause > > M> +# the build to break on other systems that don't have that tool. For now we > > M> +# still allow using the old behaviour (inheriting $PATH) if > > M> +# BUILD_WITH_STRICT_TMPPATH is set to 0 but this will eventually be removed. > > M> + > > M> +# Currently strict $PATH can cause build failures. Once the remaining issues > > M> +# have been resolved it will be turned on by default. > > M> +BUILD_WITH_STRICT_TMPPATH?=0 > > M> +.if defined(CROSSBUILD_HOST) > > M> +# When building on non-FreeBSD we can't rely on the tools in /usr/bin being compatible > > M> +# with what FreeBSD expects. Therefore we only use tools from STRICTTMPPATH > > M> +# during the world build stage. We build most tools during the bootstrap-tools > > M> +# phase but symlink host tools that are known to work instead of building them > > M> +BUILD_WITH_STRICT_TMPPATH:=1 > > M> +.endif > > M> +.if ${BUILD_WITH_STRICT_TMPPATH} != 0 > > M> +TMPPATH= ${STRICTTMPPATH} > > M> +.else > > M> +TMPPATH= ${STRICTTMPPATH}:${PATH} > > M> +.endif > > M> + > > M> +# > > M> +# Avoid running mktemp(1) unless actually needed. > > M> +# It may not be functional, e.g., due to new ABI > > M> +# when in the middle of installing over this system. > > M> +# > > M> +.if make(distributeworld) || make(installworld) || make(stageworld) > > M> +.if ${BUILD_WITH_STRICT_TMPPATH} != 0 > > M> +MKTEMP=${WORLDTMP}/legacy/usr/bin/mktemp > > M> +.if !exists(${MKTEMP}) > > M> +.error mktemp binary doesn't exist in expected location: ${MKTEMP} > > M> +.endif > > M> +.else > > M> +MKTEMP=mktemp > > M> +.endif > > M> +INSTALLTMP!= ${MKTEMP} -d -u -t install > > M> + > > M> +.if ${.MAKE.OS} == "FreeBSD" > > M> +# When building on FreeBSD we always copy the host tools instead of linking > > M> +# into INSTALLTMP to avoid issues with incompatible libraries (see r364030). > > M> +# Note: we could create links if we don't intend to update the current machine. > > M> +INSTALLTMP_COPY_HOST_TOOL=cp > > M> +.else > > M> +# However, this is not necessary on Linux/macOS. Additionally, copying the host > > M> +# tools to another directory with cp results in AMFI Launch Constraint > > M> +# Violations on macOS Ventura as part of its System Integrity Protection. > > M> +INSTALLTMP_COPY_HOST_TOOL=ln -s > > M> +.endif > > M> +.endif > > M> + > > M> +.if make(stagekernel) || make(distributekernel) > > M> +TAGS+= kernel > > M> +PACKAGE= kernel > > M> +.endif > > M> + > > M> +# > > M> +# Building a world goes through the following stages > > M> +# > > M> +# 1. legacy stage [BMAKE] > > M> +# This stage is responsible for creating compatibility > > M> +# shims that are needed by the bootstrap-tools, > > M> +# build-tools and cross-tools stages. These are generally > > M> +# APIs that tools from one of those three stages need to > > M> +# build that aren't present on the host. > > M> +# 1. bootstrap-tools stage [BMAKE] > > M> +# This stage is responsible for creating programs that > > M> +# are needed for backward compatibility reasons. They > > M> +# are not built as cross-tools. > > M> +# 2. build-tools stage [TMAKE] > > M> +# This stage is responsible for creating the object > > M> +# tree and building any tools that are needed during > > M> +# the build process. Some programs are listed during > > M> +# this phase because they build binaries to generate > > M> +# files needed to build these programs. This stage also > > M> +# builds the 'build-tools' target rather than 'all'. > > M> +# 3. cross-tools stage [XMAKE] > > M> +# This stage is responsible for creating any tools that > > M> +# are needed for building the system. A cross-compiler is one > > M> +# of them. This differs from build tools in two ways: > > M> +# 1. the 'all' target is built rather than 'build-tools' > > M> +# 2. these tools are installed into TMPPATH for stage 4. > > M> +# 4. world stage [WMAKE] > > M> +# This stage actually builds the world. > > M> +# 5. install stage (optional) [IMAKE] > > M> +# This stage installs a previously built world. > > M> +# > > M> + > > M> +BOOTSTRAPPING?= 0 > > M> +# Keep these in sync > > M> +MINIMUM_SUPPORTED_OSREL?= 1104001 > > M> +MINIMUM_SUPPORTED_REL?= 11.4 > > M> + > > M> +# Common environment for world related stages > > M> +CROSSENV+= \ > > M> + MACHINE_ARCH=${TARGET_ARCH} \ > > M> + MACHINE=${TARGET} \ > > M> + CPUTYPE=${TARGET_CPUTYPE} > > M> +.if ${MK_META_MODE} != "no" > > M> +# Don't rebuild build-tools targets during normal build. > > M> +CROSSENV+= BUILD_TOOLS_META=.NOMETA > > M> +.endif > > M> +.if defined(TARGET_CFLAGS) > > M> +CROSSENV+= ${TARGET_CFLAGS} > > M> +.endif > > M> +.if (${TARGET} != ${MACHINE} && !defined(WITH_LOCAL_MODULES)) || \ > > M> + defined(WITHOUT_LOCAL_MODULES) > > M> +CROSSENV+= LOCAL_MODULES= > > M> +.endif > > M> + > > M> +BOOTSTRAPPING_OSRELDATE?=${OSRELDATE} > > M> + > > M> +# bootstrap-tools stage > > M> +BMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \ > > M> + TOOLS_PREFIX=${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \ > > M> + PATH=${BPATH:Q}:${PATH:Q} \ > > M> + WORLDTMP=${WORLDTMP} \ > > M> + MAKEFLAGS="-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}" > > M> +# need to keep this in sync with targets/pseudo/bootstrap-tools/Makefile > > M> +BSARGS= DESTDIR= \ > > M> + OBJTOP='${WORLDTMP}/obj-tools' \ > > M> + OBJROOT='$${OBJTOP}/' \ > > M> + UNIVERSE_TOOLCHAIN_PATH=${UNIVERSE_TOOLCHAIN_PATH} \ > > M> + MAKEOBJDIRPREFIX= \ > > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \ > > M> + BWPHASE=${.TARGET:C,^_,,} \ > > M> + -DNO_CPU_CFLAGS \ > > M> + -DNO_LINT \ > > M> + -DNO_PIC \ > > M> + -DNO_SHARED \ > > M> + MK_ASAN=no \ > > M> + MK_CTF=no \ > > M> + MK_CLANG_EXTRAS=no \ > > M> + MK_CLANG_FORMAT=no \ > > M> + MK_CLANG_FULL=no \ > > M> + MK_HTML=no \ > > M> + MK_MAN=no \ > > M> + MK_PROFILE=no \ > > M> + MK_RETPOLINE=no \ > > M> + MK_SSP=no \ > > M> + MK_TESTS=no \ > > M> + MK_UBSAN=no \ > > M> + MK_WERROR=no \ > > M> + MK_INCLUDES=yes \ > > M> + MK_MAN_UTILS=yes > > M> + > > M> +BMAKE= \ > > M> + ${TIME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \ > > M> + ${BSARGS} > > M> +.if empty(.MAKEOVERRIDES:MMK_LLVM_TARGET_ALL) > > M> +BMAKE+= MK_LLVM_TARGET_ALL=no > > M> +.endif > > M> + > > M> +# build-tools stage > > M> +TMAKE= \ > > M> + ${TIME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \ > > M> + TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \ > > M> + DESTDIR= \ > > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \ > > M> + BWPHASE=${.TARGET:C,^_,,} \ > > M> + -DNO_CPU_CFLAGS \ > > M> + -DNO_LINT \ > > M> + MK_ASAN=no \ > > M> + MK_CTF=no \ > > M> + MK_CLANG_EXTRAS=no \ > > M> + MK_CLANG_FORMAT=no \ > > M> + MK_CLANG_FULL=no \ > > M> + MK_LLDB=no \ > > M> + MK_RETPOLINE=no \ > > M> + MK_SSP=no \ > > M> + MK_TESTS=no \ > > M> + MK_UBSAN=no \ > > M> + MK_WERROR=no > > M> + > > M> +# cross-tools stage > > M> +# TOOLS_PREFIX set in BMAKE > > M> +XMAKE= ${BMAKE} \ > > M> + TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} \ > > M> + MK_LLDB=no \ > > M> + MK_LLVM_BINUTILS=no \ > > M> + MK_TESTS=no > > M> + > > M> +# kernel-tools stage > > M> +KTMAKEENV= INSTALL="sh ${.CURDIR}/tools/install.sh" \ > > M> + PATH=${BPATH:Q}:${PATH:Q} \ > > M> + WORLDTMP=${WORLDTMP} \ > > M> + MAKEFLAGS="-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}" > > M> + > > M> +KTMAKE= ${TIME_ENV} \ > > M> + TOOLS_PREFIX=${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \ > > M> + ${KTMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \ > > M> + DESTDIR= \ > > M> + OBJTOP='${WORLDTMP}/obj-kernel-tools' \ > > M> + OBJROOT='$${OBJTOP}/' \ > > M> + UNIVERSE_TOOLCHAIN_PATH=${UNIVERSE_TOOLCHAIN_PATH} \ > > M> + MAKEOBJDIRPREFIX= \ > > M> + BOOTSTRAPPING=${BOOTSTRAPPING_OSRELDATE} \ > > M> + -DNO_CPU_CFLAGS \ > > M> + -DNO_LINT \ > > M> + -DNO_PIC \ > > M> + -DNO_SHARED \ > > M> + MK_CTF=no \ > > M> + MK_HTML=no \ > > M> + MK_MAN=no \ > > M> + MK_PROFILE=no \ > > M> + MK_SSP=no \ > > M> + MK_RETPOLINE=no \ > > M> + MK_WERROR=no > > M> + > > M> +# world stage > > M> +WMAKEENV= ${CROSSENV} \ > > M> + INSTALL="${INSTALL_CMD} -U" \ > > M> + PATH=${TMPPATH:Q} \ > > M> + SYSROOT=${WORLDTMP} > > M> + > > M> +# make hierarchy > > M> *** 5431 LINES SKIPPED *** > > > > -- > Alexander Motin > --06e1cace14e44ba6b615e2ddd8eabb7c Content-Type: text/html Content-Transfer-Encoding: quoted-printable

=
On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote:<= br>
Hi Gleb,

It is the first time I see a panic like this.=   I'll think about it 
tomorrow.  But I'd a= ppreciate any information on what is your workload 
a= nd what are you doing related to ZIL (O_SYNC, fsync(), sync=3Dalways,&nb= sp;
etc) to trigger it?  What is your pool configurat= ion?

I'm not Gleb, but this wa= s something at $WORK, so I can perhaps help.  I've included the out= put of zpool status, and all non-default settings in the zpool.  No= te that we don't use a ZIL device.

Thanks,<= br>

Drew


# zpool status
  pool: zroot
stat= e: ONLINE
status: Some supported and requested features ar= e not enabled on the pool.
The pool can still be used, but= some features are unavailable.
action: Enable all feature= s using 'zpool upgrade'. Once this is done,
the pool may n= o longer be accessible by software that does not support
t= he features. See zpool-features(7) for details.
config:

NAME       = ; STATE     READ WRITE CKSUM
zroot&nbs= p;      ONLINE     &nb= sp; 0     0     0
=   raidz1-0  ONLINE       0 =     0     0
  = ;  nda0p2  ONLINE       0 &= nbsp;   0     0
  =   nda1p2  ONLINE       0 &n= bsp;   0     0
  &= nbsp; nda2p2  ONLINE       0 &nb= sp;   0     0
spares
  nda3p2    AVAIL   
=

errors: No known data errors

# zpool get all | grep -v default
NAME   P= ROPERTY           = ;            VALU= E            = ;            = ;            = ;            = ;           SOURCE
=
zroot  size       &nb= sp;           &nb= sp;       149G    &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;       -
zroot  capac= ity           &nb= sp;           14% = ;            = ;            = ;            = ;            = ;            -
zroot  health       =             =       ONLINE      = ;            = ;            = ;            = ;            = ;    -
zroot  guid   &n= bsp;           &n= bsp;           5099990= 586719219765          =             =             =            -
=
zroot  bootfs        &= nbsp;           &= nbsp;    zroot/ROOT/NCD-7827-freebsd-2023-08-29-merge-oca= -r202308291935  local
zroot  dedupratio &nb= sp;           &nb= sp;       1.00x    &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;      -
zroot  free =             =             =   127G          &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp; -
zroot  allocated     = ;            = ;     22.3G       = ;            = ;            = ;            = ;            = ;    -
zroot  readonly  &nbs= p;           &nbs= p;        off    =             =             =             =             =          -
zroot&n= bsp; expandsize         &nb= sp;           - &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp; -
zroot  freeing     &= nbsp;           &= nbsp;      0      = ;            = ;            = ;            = ;            = ;         -
zroot&= nbsp; fragmentation         = ;         28%   &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;         -
zr= oot  leaked         &n= bsp;           &n= bsp;   0         =             =             =             =             =       -
zroot  checkpoint&nb= sp;           &nb= sp;        -    &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;          -
<= div>zroot  load_guid        = ;            = ;  7612195721260396606       &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp; -
zroot  bcloneused     =             =     0        &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;      -
zroot  bclonesaved=             =         0    &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;          -
zroot  bcloneratio        =             1.00x=             =             =             =             =            -
=
zroot  feature@async_destroy     &nbs= p;    enabled       &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp; local
zroot  feature@empty_bpobj  &nbs= p;         active  &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;       local
zroot  = feature@lz4_compress        &nbs= p;  active         &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;            lo= cal
zroot  feature@multi_vdev_crash_dump  enable= d            = ;            = ;            = ;            = ;         local
zr= oot  feature@spacemap_histogram     active = ;            = ;            = ;            = ;            = ;         local
zr= oot  feature@enabled_txg       &= nbsp;    active       =             =             =             =             =    local
zroot  feature@hole_birth &nb= sp;           active&n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;         local
zroot  feature@extensible_dataset     active&n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;         local
zroot  feature@embedded_data      &n= bsp;   active        &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp; local
zroot  feature@bookmarks   = ;           enabled&nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;        local
zroot&= nbsp; feature@filesystem_limits      enabled&nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;        local
zroot&= nbsp; feature@large_blocks       &nbs= p;   enabled        &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;            l= ocal
zroot  feature@large_dnode   &nbs= p;        enabled   &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;     local
zroot  feature@sha= 512           &nb= sp;     enabled      &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;  local
zroot  feature@skein  &nb= sp;           &nb= sp;   enabled        &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;            = local
zroot  feature@edonr    &nb= sp;           &nb= sp; disabled          =             =             =             =           local
zroot  feature@userobj_accounting     active=             =             =             =             =           local
zroot  feature@encryption      &nb= sp;      enabled     &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;   local
zroot  feature@project_quota&= nbsp;         active  =             =             =             =             =         local
zroot&nbs= p; feature@device_removal        = ; enabled          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;          local
zroot  feature@obsolete_counts     &= nbsp;  enabled         = ;            = ;            = ;            = ;            loca= l
zroot  feature@zpool_checkpoint   &n= bsp;   enabled        =             =             =             =             = local
zroot  feature@spacemap_v2   &n= bsp;        active   &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;      local
zroot  feat= ure@allocation_classes     enabled   =             =             =             =             =       local
zroot  feature@r= esilver_defer         enabled&nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;        local
zroot&= nbsp; feature@bookmark_v2        = ;    enabled       &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp; local
zroot  feature@redaction_bookmarks &nb= sp;  enabled         &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           local<= br>
zroot  feature@redacted_datasets   &nb= sp;  enabled         &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           local<= br>
zroot  feature@bookmark_written   &nbs= p;   enabled        &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;            l= ocal
zroot  feature@log_spacemap   &nb= sp;       active    &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;     local
zroot  feature@liv= elist           &= nbsp;   enabled        = ;            = ;            = ;            = ;            = ; local
zroot  feature@device_rebuild  &nbs= p;      enabled     &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;           &n= bsp;   local
zroot  feature@zstd_compress&n= bsp;         enabled  =             =             =             =             =        local
zroot  fea= ture@draid          &n= bsp;       enabled    =             =             =             =             =      local
zroot  feature@zilsaxa= ttr           &nb= sp; disabled          =             =             =             =           local
zroot  feature@head_errlog      &n= bsp;     disabled      = ;            = ;            = ;            = ;            = ;  local
zroot  feature@blake3   =             =   disabled         &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;          local
zroot  feature@block_cloning     &nb= sp;    disabled       =             =             =             =             = local
zroot  feature@vdev_zaps_v2   &= nbsp;       disabled   &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;    local


On 30.08.2023 20:09, Gleb = Smirnoff wrote:
>    Hi Martin,
> 
> we hit this panic with this update:<= br>
> 
> panic: VERIFY(avl_is_empty(&= amp;lwb->lwb_vdev_tree)) failed
> 
> cpuid =3D 8
> time =3D 1693432004
= > KDB: stack backtrace:
> db_trace_self_wrapper() at= db_trace_self_wrapper+0x2b/frame 0xfffffe09519aa800
> = vpanic() at vpanic+0x132/frame 0xfffffe09519aa930
> spl= _panic() at spl_panic+0x3a/frame 0xfffffe09519aa990
> z= il_free_lwb() at zil_free_lwb+0x1de/frame 0xfffffe09519aa9c0
> zil_sync() at zil_sync+0x3cb/frame 0xfffffe09519aaaa0
<= div>> [Wed Aug 30 21:46:44 2023]dmu_objset_sync() at dmu_objset_sync+= 0x51b/frame 0xfffffe09519aab70
> dsl_pool_sync() at dsl= _pool_sync+0x11d/frame 0xfffffe09519aabf0
> spa_sync() = at spa_sync+0xc68/frame 0xfffffe09519aae20
> txg_sync_t= hread() at txg_sync_thread+0x2eb/frame 0xfffffe09519aaef0
= > fork_exit() at fork_exit+0x82/frame 0xfffffe09519aaf30
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe09519aaf30=
> --- trap 0x26d90a2f, rip =3D 0xbba54c81b25fd8d4, rsp= =3D 0xff60c2f6f9874bcd, rbp =3D 0x37940e3e9793bd40 --
>= ; 
> I have two cores dumped for this panic.
> 
> Our previous version is based on = 9c0a2d522f3a18fc795a7dd6266d2be2aff4b084, which
> I gue= ss has the most recent ZFS merge openzfs/zfs@009d3288d.
&g= t; 
> On Sun, Aug 27, 2023 at 05:09:06AM +0000, Ma= rtin Matuska wrote:
> M> The branch main has been up= dated by mm:
> M>
> M>
> M> commit 315ee00fa9616b0a192b6834911f98bcf5316a6b
> M> Merge: b3a714652ff0 804414aad224
> M&= gt; Author:     Martin Matuska <mm@FreeBSD.org>
> M> AuthorD= ate: 2023-08-26 21:20:04 +0000
> M> Commit: &nb= sp;   Martin Matuska <mm@= FreeBSD.org>
> M> CommitDate: 2023-08-26 21:5= 1:42 +0000
> M>
> M>  =    zfs: merge openzfs/zfs@804414aad
> M>
> M>     Notable upstream pull re= quest merges:
> M>     &nbs= p; #15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
> = M>       #15029 Do not request data L1 = buffers on scan prefetch
> M>    = ;   #15036 FreeBSD: catch up to __FreeBSD_version 1400093
<= /div>
> M>       #15039 Fix raw = receive with different indirect block size
> M> = ;      #15047 FreeBSD: Fix build on stable/13 a= fter 1302506
> M>      = ; #15049 Fix the ZFS checksum error histograms with larger record sizes<= br>
> M>       #15052 Redu= ce bloat in ereport.fs.zfs.checksum events
> M> = ;      #15056 Avoid extra snprintf() in dsl_dea= dlist_merge()
> M>     &nbs= p; #15061 Ignore pool ashift property during vdev attachment
> M>       #15063 Don't panic if = setting vdev properties is unsupported for this vdev type
= > M>       #15067 spa_min_alloc shou= ld be GCD, not min
> M>     = ;  #15071 Add explicit prefetches to bpobj_iterate()
= > M>       #15072 Adjust prefetch pa= rameters
> M>       #1= 5076 Refactor dmu_prefetch()
> M>   &= nbsp;   #15079 set autotrim default to 'off' everywhere
> M>       #15080 ZIL: Fix c= onfig lock deadlock
> M>    &nbs= p;  #15088 metaslab: tuneable to better control force ganging
> M>       #15096 Avoid wai= ting in dmu_sync_late_arrival()
> M>  &nbs= p;    #15097 BRT should return EOPNOTSUPP
&= gt; M>       #15103 Remove zl_issuer_lo= ck from zil_suspend()
> M>    &n= bsp;  #15107 Remove fastwrite mechanism
> M>&nb= sp;      #15113 libzfs: sendrecv: send_progress= _thread: handle SIGINFO/SIGUSR1
> M>  &nbs= p;    #15122 ZIL: Second attempt to reduce scope of zl_is= suer_lock
> M>       #= 15129 zpool_vdev_remove() should handle EALREADY error return
<= div>> M>       #15132 ZIL: Replay bl= ocks without next block pointer
> M>  &nbs= p;    #15148 zfs_clone_range should return descriptive er= ror codes
> M>       #= 15153 ZIL: Avoid dbuf_read() before dmu_sync()
> M>&= nbsp;      #15172 copy_file_range: fix fallback= when source create on same txg
> M>  &nbs= p;    #15180 Update outdated assertion from zio_write_com= press
> M>
> M>   = ;  Obtained from:  OpenZFS
> M>  = ;   OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13
> M>
> M>  Makefile.inc1&nbs= p;           &nbs= p;           &nbs= p;           &nbs= p; |   5 +-
> M>  cddl/lib/libzfs/Mak= efile           &= nbsp;           &= nbsp;   |   1 +
> M>  cddl/= lib/libzfs/Makefile.depend       &nbs= p;            |&n= bsp;  1 +
> M>  rescue/rescue/Makefile&nbs= p;           &nbs= p;           &nbs= p;    |   2 +-
> M>  r= escue/rescue/Makefile.depend       &n= bsp;           &n= bsp;  |   1 +
> M>  share/mk/src= .libnames.mk          =             =      |   2 +-
> M>&nbs= p; sys/contrib/openzfs/META       &nb= sp;           &nb= sp;       |   6 +-
> M>  sys/contrib/openzfs/cmd/zdb/zdb.c   &nb= sp;           &nb= sp;  |  77 ++-
> M>  sys/contrib/open= zfs/cmd/zed/agents/zfs_mod.c       | =   2 -
> M>  sys/contrib/openzfs/cmd/zed/ag= ents/zfs_retire.c    |   5 +
>= M>  .../openzfs/cmd/zed/zed.d/statechange-slot_off.sh  |&n= bsp; 61 ++
> M>  sys/contrib/openzfs/cmd/zed/ze= d.d/zed.rc           |=    5 +
> M>  sys/contrib/openzfs/cmd/= ztest.c           = ;         |   2 +-
=
> M>  sys/contrib/openzfs/config/Shellcheck.am&nbs= p;          |   3= +-
> M>  sys/contrib/openzfs/config/kernel-blk= dev.m4        | 138 +++-
> M>  .../config/kernel-block-device-operations.m4 &nb= sp;     |  35 +-
> M> = .../openzfs/config/kernel-filemap-splice-read.m4   |  25= +
> M>  .../openzfs/config/kernel-register_sys= ctl_table.m4 |  27 +
> M>  .../config/kern= el-vfs-extended-file_range.m4       | = ; 50 ++
> M>  .../openzfs/config/kernel-vfs-fil= e_range.m4        | 164 +++++
> M>  sys/contrib/openzfs/config/kernel-vfs-iov_iter.m= 4  |  26 +-
> M>  sys/contrib/openzfs= /config/kernel.m4         &= nbsp;     |  16 +
> M> = ; sys/contrib/openzfs/contrib/debian/changelog    &n= bsp;  |   6 +
> M>  .../contrib/= debian/openzfs-zfs-test.install       = ; |   2 -
> M>  .../contrib/debian/op= enzfs-zfsutils.install        | =   2 +
> M>  .../contrib/dracut/90zfs/zfs-e= nv-bootfs.service.in |  11 +-
> M>  .../dr= acut/90zfs/zfs-rollback-bootfs.service.in    |  = ; 2 +-
> M>  .../openzfs/include/os/freebsd/spl= /sys/vnode.h     |   6 +-
&g= t; M>  .../include/os/linux/kernel/linux/blkdev_compat.h  |=   17 +
> M>  .../openzfs/include/os/linux/= spl/sys/kmem_cache.h  |   8 +
> M>&nb= sp; .../openzfs/include/os/linux/spl/sys/types.h    =    |   2 +-
> M>  sys/contr= ib/openzfs/include/os/linux/spl/sys/uio.h |  12 +
>= ; M>  sys/contrib/openzfs/include/os/linux/zfs/sys/zpl.h | = 53 ++
> M>  sys/contrib/openzfs/include/sys/bp= obj.h            = |   2 +-
> M>  sys/contrib/openzfs/in= clude/sys/brt.h         &nb= sp;    |   1 +
> M>  s= ys/contrib/openzfs/include/sys/dmu.h      =         |   5 +
> M>  sys/contrib/openzfs/include/sys/dmu_impl.h &nbs= p;       |   2 -
&= gt; M>  sys/contrib/openzfs/include/sys/dmu_zfetch.h  =      |   2 -
> M> = ; sys/contrib/openzfs/include/sys/fm/fs/zfs.h    &nb= sp;   |   4 -
> M>  sys/con= trib/openzfs/include/sys/metaslab.h      &= nbsp;  |   3 -
> M>  sys/contrib= /openzfs/include/sys/metaslab_impl.h    |   6 += -
> M>  sys/contrib/openzfs/include/sys/spa_imp= l.h         |   1 +
> M>  sys/contrib/openzfs/include/sys/vdev_impl.h=         |   2 +-
<= div>> M>  sys/contrib/openzfs/include/sys/zil_impl.h &nb= sp;       |  44 +-
>= M>  sys/contrib/openzfs/include/sys/zio.h   &nbs= p;          |   1= -
> M>  sys/contrib/openzfs/include/sys/zio_ch= ecksum.h     |   2 -
> M&= gt;  sys/contrib/openzfs/lib/libzfs/Makefile.am   &n= bsp;     |   2 +-
> M>=   sys/contrib/openzfs/lib/libzfs/libzfs_pool.c   &nb= sp;   |   6 +
> M>  sys/con= trib/openzfs/lib/libzfs/libzfs_sendrecv.c   |  95 ++-
=
> M>  sys/contrib/openzfs/man/man4/zfs.4 &nbs= p;           &nbs= p;   |  10 +-
> M>  sys/contrib/= openzfs/man/man8/zfs-send.8       &nb= sp;    |  18 +-
> M>  sys/c= ontrib/openzfs/man/man8/zpool-events.8     &nbs= p;  |  23 +-
> M>  sys/contrib/openzf= s/module/Kbuild.in         =       |   1 +
> M>= ;  .../openzfs/module/os/freebsd/zfs/zfs_vnops_os.c   |&n= bsp; 10 +-
> M>  sys/contrib/openzfs/module/os/= linux/spl/spl-proc.c |  26 +-
> M>  .../op= enzfs/module/os/linux/zfs/vdev_disk.c      = ;  |  65 +-
> M>  .../openzfs/module/= os/linux/zfs/zfs_ctldir.c       |  26= +-
> M>  .../openzfs/module/os/linux/zfs/zfs_v= fsops.c       |   7 +
<= div>> M>  .../openzfs/module/os/linux/zfs/zfs_vnops_os.c = ;    |   9 +-
> M>  ..= ./openzfs/module/os/linux/zfs/zfs_znode.c     &= nbsp;  |   8 +
> M>  .../openzfs= /module/os/linux/zfs/zpl_ctldir.c       |&= nbsp;  2 +-
> M>  sys/contrib/openzfs/modu= le/os/linux/zfs/zpl_file.c |  44 +-
> M>  = .../openzfs/module/os/linux/zfs/zpl_file_range.c   | 272 +++++= +++
> M>  .../openzfs/module/os/linux/zfs/zpl_s= uper.c        |  39 +-
=
> M>  sys/contrib/openzfs/module/os/linux/zfs/zvol_os.c&= nbsp; |  28 +-
> M>  sys/contrib/openzfs/m= odule/zfs/bpobj.c         &= nbsp;   |  49 +-
> M>  sys/contr= ib/openzfs/module/zfs/brt.c       &nb= sp;       |  43 +-
>= M>  sys/contrib/openzfs/module/zfs/dbuf.c   &nbs= p;          |  11 +-
> M>  sys/contrib/openzfs/module/zfs/dmu.c =             =   | 127 ++--
> M>  sys/contrib/openzfs/mod= ule/zfs/dmu_recv.c         = |  22 +-
> M>  sys/contrib/openzfs/module= /zfs/dmu_zfetch.c        |  12 += -
> M>  sys/contrib/openzfs/module/zfs/dnode.c&= nbsp;            = |  31 +-
> M>  sys/contrib/openzfs/module/= zfs/dsl_deadlist.c      |  14 +-
=
> M>  sys/contrib/openzfs/module/zfs/dsl_scan.c &nb= sp;        |  17 +-
> M>  sys/contrib/openzfs/module/zfs/metaslab.c  =         |  83 +--
= > M>  sys/contrib/openzfs/module/zfs/spa_log_spacemap.c = |   4 +-
> M>  sys/contrib/openzfs/m= odule/zfs/spa_misc.c        &nbs= p; |   1 +
> M>  sys/contrib/openzfs/= module/zfs/vdev.c         &= nbsp;    |  90 ++-
> M>  sy= s/contrib/openzfs/module/zfs/vdev_indirect.c     |&n= bsp;  2 +-
> M>  sys/contrib/openzfs/modul= e/zfs/vdev_raidz.c        | &nbs= p; 2 +-
> M>  sys/contrib/openzfs/module/zfs/zf= s_fm.c           = |  33 +-
> M>  sys/contrib/openzfs/module= /zfs/zfs_vnops.c         | = 46 +-
> M>  sys/contrib/openzfs/module/zfs/zil= .c           &nbs= p;   | 719 +++++++++------------
> M> = ; sys/contrib/openzfs/module/zfs/zio.c     &nbs= p;         |  45 +-
> M>  sys/contrib/openzfs/module/zfs/zio_checksum.c&nb= sp;     |   2 -
> M>&n= bsp; sys/contrib/openzfs/module/zfs/zvol.c     =          |   8 +-
<= /div>
> M>  sys/contrib/openzfs/tests/runfiles/linux.run&= nbsp;      |  11 +
> M>= ;  .../openzfs/tests/test-runner/bin/zts-report.py.in |  18 +<= br>
> M>  sys/contrib/openzfs/tests/zfs-tests/cmd/.= gitignore |   1 +
> M>  .../openzfs/t= ests/zfs-tests/cmd/Makefile.am       = |   1 +
> M>  .../openzfs/tests/zfs-= tests/cmd/clonefile.c        | 333 ++= ++++++++
> M>  sys/contrib/openzfs/tests/zfs-te= sts/cmd/readmmap.c |   1 +
> M>  .../= openzfs/tests/zfs-tests/include/commands.cfg   |   1= +
> M>  .../openzfs/tests/zfs-tests/tests/Make= file.am      |  14 +
> M&= gt;  .../functional/block_cloning/block_cloning.kshlib  | = ; 54 ++
> M>  .../block_cloning/block_cloning_c= opyfilerange.ksh  |  60 ++
> M>  .../= block_cloning_copyfilerange_cross_dataset.ksh  |  65 ++
> M>  .../block_cloning_copyfilerange_fallback.ksh&nb= sp;      |  86 +++
> M>= ;  ...ock_cloning_copyfilerange_fallback_same_txg.ksh |  66 ++=
> M>  .../block_cloning_copyfilerange_partial.= ksh        |  68 ++
> M>  .../block_cloning_disabled_copyfilerange.ksh &nb= sp;     |  60 ++
> M> = .../block_cloning_disabled_ficlone.ksh     &nb= sp;       |  50 ++
>= M>  .../block_cloning_disabled_ficlonerange.ksh  &nbs= p;     |  50 ++
> M>  = .../block_cloning/block_cloning_ficlone.ksh     = ;   |  56 ++
> M>  .../block_clo= ning/block_cloning_ficlonerange.ksh   |  56 ++
<= div>> M>  .../block_cloning_ficlonerange_partial.ksh &nb= sp;       |  64 ++
>= M>  .../tests/functional/block_cloning/cleanup.ksh  &= nbsp;  |  34 +
> M>  .../tests/functi= onal/block_cloning/setup.ksh       | = 36 ++
> M>  .../cli_root/zpool_attach/attach-o= _ashift.ksh      |  30 +-
&g= t; M>  .../cli_root/zpool_replace/replace-o_ashift.ksh &nbs= p;  |  32 +-
> M>  .../cli_root/zpool= _replace/replace_prop_ashift.ksh |  24 +-
> M>&= nbsp; sys/modules/zfs/zfs_config.h      &n= bsp;           &n= bsp;    |  53 +-
> M>  sys/= modules/zfs/zfs_gitrev.h        =             =    |   2 +-
> M>  113 files= changed, 3322 insertions(+), 906 deletions(-)
> M><= br>
> M> diff --cc Makefile.inc1
> M>= ; index 8838d31c795a,000000000000..f740cc5abcca
> M>= mode 100644,000000..100644
> M> --- a/Makefile.inc1=
> M> +++ b/Makefile.inc1
> M> @= @@ -1,3692 -1,0 +1,3695 @@@
> M>  +#
<= div>> M>  +#
> M>  +# Make command l= ine options:
> M>  +# -DNO_CLEANDIR run ${MAKE}= clean, instead of ${MAKE} cleandir
> M>  +# -D= NO_CLEAN do not clean at all
> M>  +# -DDB_FROM= _SRC use the user/group databases in src/etc instead of
&g= t; M>  +#     the system database when installing= .
> M>  +# -DNO_SHARE do not go into share subd= ir
> M>  +# -DKERNFAST define NO_KERNEL{CONFIG,= CLEAN,OBJ}
> M>  +# -DNO_KERNELCONFIG do not ru= n config in ${MAKE} buildkernel
> M>  +# -DNO_K= ERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel
> M>  +# -DNO_KERNELOBJ do not run ${MAKE} obj in ${MAKE} bu= ildkernel
> M>  +# -DNO_ROOT install without us= ing root privilege
> M>  +# -DWITHOUT_CTF do no= t run the DTrace CTF conversion tools on built objects
>= ; M>  +# LOCAL_DIRS=3D"list of dirs" to add additional dirs to t= he SUBDIR list
> M>  +# LOCAL_ITOOLS=3D"list of= tools" to add additional tools to the ITOOLS list
> M&= gt;  +# LOCAL_LIB_DIRS=3D"list of dirs" to add additional dirs to l= ibraries target
> M>  +# LOCAL_MTREE=3D"list of= mtree files" to process to allow local directories
> M= >  +#     to be created before files are installe= d
> M>  +# LOCAL_LEGACY_DIRS=3D"list of dirs" t= o add additional dirs to the legacy
> M>  +# &n= bsp;   target
> M>  +# LOCAL_BSTOOL_D= IRS=3D"list of dirs" to add additional dirs to the
> M&= gt;  +#     bootstrap-tools target
>= ; M>  +# LOCAL_TOOL_DIRS=3D"list of dirs" to add additional dirs= to the build-tools
> M>  +#    = target
> M>  +# LOCAL_XTOOL_DIRS=3D"list of di= rs" to add additional dirs to the
> M>  +# &nbs= p;   cross-tools target
> M>  +# META= LOG=3D"path to metadata log" to write permission and ownership
=
> M>  +#     when NO_ROOT is set.  (= default: ${DESTDIR}/${DISTDIR}/METALOG,
> M>  += #           check /etc= /make.conf for DISTDIR)
> M>  +# TARGET=3D"mach= ine" to crossbuild world for a different machine type
>= M>  +# TARGET_ARCH=3D may be required when a TARGET supports mu= ltiple endians
> M>  +# BUILDENV_SHELL=3D shell= to launch for the buildenv target (def:${SHELL})
> M&g= t;  +# WORLD_FLAGS=3D additional flags to pass to make(1) during bu= ildworld
> M>  +# KERNEL_FLAGS=3D additional fl= ags to pass to make(1) during buildkernel
> M> = +# SUBDIR_OVERRIDE=3D"list of dirs" to build rather than everything.
> M>  +#     All libraries and inc= ludes, and some build tools will still build.
> M>&n= bsp; +
> M>  +#
> M>  = +# The intended user-driven targets are:
> M>  = +# buildworld  - rebuild *everything*, including glue to help do up= grades
> M>  +# installworld- install everythin= g built by "buildworld"
> M>  +# checkworld&nbs= p; - run test suite on installed world
> M>  +#= doxygen     - build API documentation of the kernel=
> M>  +#
> M>  +# Sta= ndard targets (not defined here) are documented in the makefiles in
<= /div>
> M>  +# /usr/share/mk.  These include:
> M>  +# obj depend all install clean cleandepend cl= eanobj
> M>  +
> M>  += .if !defined(TARGET) || !defined(TARGET_ARCH)
> M>&n= bsp; +.error Both TARGET and TARGET_ARCH must be defined.
= > M>  +.endif
> M>  +
= > M>  +.if make(showconfig) || make(test-system-*)
<= div>> M>  +_MKSHOWCONFIG=3D t
> M>  = +.endif
> M>  +
> M>  = +SRCDIR?=3D ${.CURDIR}
> M>  +LOCALBASE?=3D /us= r/local
> M>  +TIME_ENV ?=3D time env
=
> M>  +
> M>  +.include "share/= mk/src.tools.mk"
> M>  +
> M&g= t;  +# Cross toolchain changes must be in effect before bsd.compile= r.mk
> M>  +# so that gets the right CC, and pa= ss CROSS_TOOLCHAIN to submakes.
> M>  +.if defi= ned(CROSS_TOOLCHAIN)
> M>  +.if exists(${LOCALB= ASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk)
> M>&nb= sp; +.include "${LOCALBASE}/share/toolchains/${CROSS_TOOLCHAIN}.mk"
<= /div>
> M>  +.elif exists(${CROSS_TOOLCHAIN})
> M>  +.include "${CROSS_TOOLCHAIN}"
> M&= gt;  +.else
> M>  +.error CROSS_TOOLCHAIN = ${CROSS_TOOLCHAIN} not found
> M>  +.endif
<= /div>
> M>  +CROSSENV+=3DCROSS_TOOLCHAIN=3D"${CROSS_TOOLC= HAIN}"
> M>  +.elif defined(UNIVERSE_TOOLCHAIN)=
> M>  +UNIVERSE_TOOLCHAIN_PATH?=3D${HOST_OBJTO= P}/tmp/usr/bin
> M>  +XCC?=3D"${UNIVERSE_TOOLCH= AIN_PATH}/cc"
> M>  +XCXX?=3D"${UNIVERSE_TOOLCH= AIN_PATH}/c++"
> M>  +XCPP?=3D"${UNIVERSE_TOOLC= HAIN_PATH}/cpp"
> M>  +XLD?=3D"${UNIVERSE_TOOLC= HAIN_PATH}/ld"
> M>  +.endif
>= M>  +.if defined(CROSS_TOOLCHAIN_PREFIX)
> M&g= t;  +CROSS_COMPILER_PREFIX?=3D${CROSS_TOOLCHAIN_PREFIX}
> M>  +.endif
> M>  +
> M>  +XCOMPILERS=3D CC CXX CPP
> M>&n= bsp; +.for COMPILER in ${XCOMPILERS}
> M>  +.if= defined(CROSS_COMPILER_PREFIX)
> M>  +X${COMPI= LER}?=3D ${CROSS_COMPILER_PREFIX}${${COMPILER}}
> M>=   +.else
> M>  +X${COMPILER}?=3D ${${COMPI= LER}}
> M>  +.endif
> M>&nb= sp; +.endfor
> M>  +# If a full path to an exte= rnal cross compiler is given, don't build
> M> = +# a cross compiler.
> M>  +.if ${XCC:N${CCACH= E_BIN}:M/*}
> M>  +MK_CLANG_BOOTSTRAP=3D no
=
> M>  +# Make sure sub-makes see the option as dis= abled so the hack in bsd.sys.mk to
> M>  +# wor= k around incompatible headers in Clang's resource directory is enabled.<= br>
> M>  +.MAKEOVERRIDES+=3D MK_CLANG_BOOTSTRAP
> M>  +.endif
> M>  +
> M>  +# Pull in compiler metadata from buildworl= d/toolchain if possible to avoid
> M>  +# runni= ng CC from bsd.compiler.mk.
> M>  +.if make(ins= tallworld) || make(install) || make(distributeworld) || \
= > M>  +    make(stageworld)
>= M>  +.-include "${OBJTOP}/toolchain-metadata.mk"
= > M>  +.if !defined(_LOADED_TOOLCHAIN_METADATA)
> M>  +.error A build is required first.  You may have = the wrong MAKEOBJDIRPREFIX set.
> M>  +.endif
> M>  +.endif
> M>  +
> M>  +# Pull in COMPILER_TYPE and COMPILER_FREE= BSD_VERSION early. Pull it from the
> M>  +# tr= ee to be friendlier to foreign OS builds. It's safe to do so uncondition= ally
> M>  +# here since we will always have th= e right make, unlike in src/Makefile
> M>  +# D= on't include bsd.linker.mk yet until XBINUTILS is handled (after src.opt= s.mk)
> M>  +_NO_INCLUDE_LINKERMK=3D t
> M>  +# We also want the X_COMPILER* variables if we a= re using an external toolchain.
> M>  +_WANT_TO= OLCHAIN_CROSS_VARS=3D t
> M>  +.include "share/= mk/bsd.compiler.mk"
> M>  +.undef _NO_INCLUDE_L= INKERMK
> M>  +.undef _WANT_TOOLCHAIN_CROSS_VAR= S
> M>  +# src.opts.mk depends on COMPILER_FEAT= URES
> M>  +.include "share/mk/src.opts.mk"
=
> M>  +
> M>  +.if ${TARG= ET} =3D=3D ${MACHINE}
> M>  +TARGET_CPUTYPE?=3D= ${CPUTYPE}
> M>  +.else
> M>= ;  +TARGET_CPUTYPE?=3D
> M>  +.endif
> M>  +.if !empty(TARGET_CPUTYPE)
>= M>  +_TARGET_CPUTYPE=3D${TARGET_CPUTYPE}
> M&g= t;  +.else
> M>  +_TARGET_CPUTYPE=3Ddummy<= br>
> M>  +.endif
> M>  +.= if ${TARGET} =3D=3D "arm"
> M>  +.if ${TARGET_C= PUTYPE:M*soft*} =3D=3D ""
> M>  +TARGET_TRIPLE_= ABI=3D gnueabihf
> M>  +.else
>= ; M>  +TARGET_TRIPLE_ABI=3D gnueabi
> M>&nbs= p; +.endif
> M>  +.endif
> M&g= t;  +MACHINE_TRIPLE_ABI?=3D unknown
> M>  = +MACHINE_TRIPLE?=3D${MACHINE_ARCH:S/amd64/x86_64/}-${MACHINE_TRIPLE_ABI}= -freebsd${OS_REVISION}
> M>  +TARGET_TRIPLE_ABI= ?=3D unknown
> M>  +TARGET_TRIPLE?=3D ${TARGET_= ARCH:S/amd64/x86_64/}-${TARGET_TRIPLE_ABI}-freebsd${OS_REVISION}
> M>  +KNOWN_ARCHES?=3D aarch64/arm64 \
= > M>  + amd64 \
> M>  + armv6/arm = \
> M>  + armv7/arm \
> M>=   + i386 \
> M>  + powerpc \
> M>  + powerpc64/powerpc \
> M> = ; + powerpc64le/powerpc \
> M>  + powerpcspe/= powerpc \
> M>  + riscv64/riscv
= > M>  +
> M>  +.if ${TARGET} =3D=3D = ${TARGET_ARCH}
> M>  +_t=3D ${TARGET}
> M>  +.else
> M>  +_t=3D ${T= ARGET_ARCH}/${TARGET}
> M>  +.endif
> M>  +.for _t in ${_t}
> M>  +.i= f empty(KNOWN_ARCHES:M${_t})
> M>  +.error Unkn= own target ${TARGET_ARCH}:${TARGET}.
> M>  +.en= dif
> M>  +.endfor
> M>&nbs= p; +
> M>  +.if ${TARGET_ARCH} =3D=3D "amd64"
> M>  +LIBCOMPAT_INCLUDE_DIRS+=3D i386
=
> M>  +.elif ${TARGET_ARCH} =3D=3D "aarch64"
> M>  +LIBCOMPAT_INCLUDE_DIRS+=3D arm
> M= >  +.endif
> M>  +
> M= >  +.if ${.MAKE.OS} !=3D "FreeBSD"
> M> = ; +CROSSBUILD_HOST=3D${.MAKE.OS}
> M>  +.if ${.= MAKE.OS} !=3D "Linux" && ${.MAKE.OS} !=3D "Darwin"
> M>  +.warning Unsupported crossbuild system: ${.MAKE.OS}. = Build will probably fail!
> M>  +.endif
> M>  +# We need to force NO_ROOT/DB_FROM_SRC builds w= hen building on other operating
> M>  +# system= s since the BSD.foo.dist specs contain users and groups that do not
<= /div>
> M>  +# exist by default on a Linux/MacOS system.<= br>
> M>  +NO_ROOT:=3D 1
> M>&n= bsp; +DB_FROM_SRC:=3D 1
> M>  +.export NO_ROOT<= br>
> M>  +.endif
> M>  +<= br>
> M>  +# If all targets are disabled for system= llvm then don't expect it to work
> M>  +# for= cross-builds.
> M>  +.if !defined(TOOLS_PREFIX= ) && ${MK_LLVM_TARGET_ALL} =3D=3D "no" && \
> M>  +    ${MACHINE} !=3D ${TARGET} &&a= mp; ${MACHINE_ARCH} !=3D ${TARGET_ARCH} && \
> = M>  +    !make(showconfig)
> M&g= t;  +MK_SYSTEM_COMPILER=3D no
> M>  +MK_SY= STEM_LINKER=3D no
> M>  +.endif
&= gt; M>  +
> M>  +# Handle external binu= tils.
> M>  +.if defined(CROSS_TOOLCHAIN_PREFIX= )
> M>  +CROSS_BINUTILS_PREFIX?=3D${CROSS_TOOLC= HAIN_PREFIX}
> M>  +.endif
> M= >  +XBINUTILS=3D AS AR ELFCTL LD NM OBJCOPY RANLIB SIZE STRINGS = STRIPBIN
> M>  +.for BINUTIL in ${XBINUTILS}
> M>  +.if defined(CROSS_BINUTILS_PREFIX) &&a= mp; \
> M>  +    exists(${CROSS_= BINUTILS_PREFIX}/${${BINUTIL}})
> M>  +X${BINUT= IL}?=3D ${CROSS_BINUTILS_PREFIX:C,/*$,,}/${${BINUTIL}}
>= ; M>  +.else
> M>  +X${BINUTIL}?=3D ${$= {BINUTIL}}
> M>  +.endif
> M&g= t;  +.endfor
> M>  +
> M&= gt;  +# If a full path to an external linker is given, don't build = lld.
> M>  +.if ${XLD:M/*}
> M= >  +MK_LLD_BOOTSTRAP=3D no
> M>  +.endi= f
> M>  +
> M>  +# We = also want the X_LINKER* variables if we are using an external toolchain.=
> M>  +_WANT_TOOLCHAIN_CROSS_VARS=3D t
> M>  +.include "share/mk/bsd.linker.mk"
> M>  +.undef _WANT_TOOLCHAIN_CROSS_VARS
> = M>  +
> M>  +# Begin WITH_SYSTEM_COMPIL= ER / WITH_SYSTEM_LD
> M>  +
> = M>  +# WITH_SYSTEM_COMPILER - Pull in needed values and make a d= ecision.
> M>  +
> M> = +# Check if there is a local compiler that can satisfy as an external c= ompiler.
> M>  +# Which compiler is expected to= be used?
> M>  +.if ${MK_CLANG_BOOTSTRAP} =3D=3D= "yes"
> M>  +WANT_COMPILER_TYPE=3D clang
> M>  +.else
> M>  +WANT_CO= MPILER_TYPE=3D
> M>  +.endif
>= M>  +
> M>  +.if !defined(WANT_COMPILE= R_FREEBSD_VERSION) && !make(showconfig) && \
> M>  +    !make(test-system-linker)
> M>  +.if ${WANT_COMPILER_TYPE} =3D=3D "clang"
> M>  +WANT_COMPILER_FREEBSD_VERSION_FILE=3D lib/cla= ng/freebsd_cc_version.h
> M>  +WANT_COMPILER_FR= EEBSD_VERSION!=3D \
> M>  + awk '$$2 =3D=3D "FR= EEBSD_CC_VERSION" {printf("%d\n", $$3)}' \
> M> = ; + ${SRCDIR}/${WANT_COMPILER_FREEBSD_VERSION_FILE} || echo unknown
<= /div>
> M>  +WANT_COMPILER_VERSION_FILE=3D lib/clang/incl= ude/clang/Basic/Version.inc
> M>  +WANT_COMPILE= R_VERSION!=3D \
> M>  + awk '$$2 =3D=3D "CLANG_= VERSION" {split($$3, a, "."); print a[1] * 10000 + a[2] * 100 + a[3]}' \=
> M>  + ${SRCDIR}/${WANT_COMPILER_VERSION_FILE= } || echo unknown
> M>  +.endif
&= gt; M>  +.export WANT_COMPILER_FREEBSD_VERSION WANT_COMPILER_VER= SION
> M>  +.endif # !defined(WANT_COMPILER_FRE= EBSD_VERSION)
> M>  +
> M>&= nbsp; +# It needs to be the same revision as we would build for the boot= strap.
> M>  +# If the expected vs CC is differ= ent then we can't skip.
> M>  +# GCC cannot be = used for cross-arch yet.  For clang we pass -target later if
> M>  +# TARGET_ARCH!=3DMACHINE_ARCH.
&= gt; M>  +.if ${MK_SYSTEM_COMPILER} =3D=3D "yes" && \
=
> M>  +    defined(WANT_COMPILER_FR= EEBSD_VERSION) && \
> M>  +  =   ${MK_CLANG_BOOTSTRAP} =3D=3D "yes" && \
>= ; M>  +    !make(xdev*) && \
> M>  +    ${X_COMPILER_TYPE} =3D=3D ${WANT_= COMPILER_TYPE} && \
> M>  +  =   (${X_COMPILER_TYPE} =3D=3D "clang" || ${TARGET_ARCH} =3D=3D ${MAC= HINE_ARCH}) && \
> M>  +  &nb= sp; ${X_COMPILER_VERSION} =3D=3D ${WANT_COMPILER_VERSION} && \
> M>  +    ${X_COMPILER_FREEBSD_V= ERSION} =3D=3D ${WANT_COMPILER_FREEBSD_VERSION}
> M>=   +# Everything matches, disable the bootstrap compiler.
<= div>> M>  +MK_CLANG_BOOTSTRAP=3D no
> M>&= nbsp; +USING_SYSTEM_COMPILER=3D yes
> M>  +.end= if # ${WANT_COMPILER_TYPE} =3D=3D ${COMPILER_TYPE}
> M&= gt;  +
> M>  +# WITH_SYSTEM_LD - Pull in n= eeded values and make a decision.
> M>  +
> M>  +# Check if there is a local linker that can s= atisfy as an external linker.
> M>  +# Which li= nker is expected to be used?
> M>  +.if ${MK_LL= D_BOOTSTRAP} =3D=3D "yes"
> M>  +WANT_LINKER_TY= PE=3D lld
> M>  +.else
> M>= ;  +WANT_LINKER_TYPE=3D
> M>  +.endif
<= /div>
> M>  +
> M>  +.if !define= d(WANT_LINKER_FREEBSD_VERSION) && !make(showconfig) && \=
> M>  +    !make(test-system-co= mpiler)
> M>  +.if ${WANT_LINKER_TYPE} =3D=3D "= lld"
> M>  +WANT_LINKER_FREEBSD_VERSION_FILE=3D= lib/clang/include/lld/Common/Version.inc
> M> = +WANT_LINKER_FREEBSD_VERSION!=3D \
> M>  + awk= '$$2 =3D=3D "LLD_FREEBSD_VERSION" {print $$3}' \
> M&g= t;  + ${SRCDIR}/${WANT_LINKER_FREEBSD_VERSION_FILE} || echo unknown=
> M>  +WANT_LINKER_VERSION_FILE=3D lib/clang/i= nclude/lld/Common/Version.inc
> M>  +WANT_LINKE= R_VERSION!=3D \
> M>  + awk '$$2 =3D=3D "LLD_VE= RSION_STRING" {gsub("\"", "", $$3); split($$3, a, "."); print a[1] * 100= 00 + a[2] * 100 + a[3]}' \
> M>  + ${SRCDIR}/${= WANT_LINKER_VERSION_FILE} || echo unknown
> M> = +.else
> M>  +WANT_LINKER_FREEBSD_VERSION_FILE= =3D
> M>  +WANT_LINKER_FREEBSD_VERSION=3D
> M>  +.endif
> M>  +.expor= t WANT_LINKER_FREEBSD_VERSION WANT_LINKER_VERSION
> M&g= t;  +.endif # !defined(WANT_LINKER_FREEBSD_VERSION)
&= gt; M>  +
> M>  +.if ${MK_SYSTEM_LINKER= } =3D=3D "yes" && \
> M>  +  =   defined(WANT_LINKER_FREEBSD_VERSION) && \
&= gt; M>  +    (${MK_LLD_BOOTSTRAP} =3D=3D "yes") &= amp;& \
> M>  +    !make(xde= v*) && \
> M>  +    ${X_= LINKER_TYPE} =3D=3D ${WANT_LINKER_TYPE} && \
> = M>  +    ${X_LINKER_VERSION} =3D=3D ${WANT_LINKER= _VERSION} && \
> M>  +   = ; ${X_LINKER_FREEBSD_VERSION} =3D=3D ${WANT_LINKER_FREEBSD_VERSION}
<= /div>
> M>  +# Everything matches, disable the bootstrap = linker.
> M>  +MK_LLD_BOOTSTRAP=3D no
=
> M>  +USING_SYSTEM_LINKER=3D yes
> M&g= t;  +.endif # ${WANT_LINKER_TYPE} =3D=3D ${LINKER_TYPE}
> M>  +
> M>  +# WITH_SYSTEM_COMP= ILER / WITH_SYSTEM_LINKER - Handle defaults and debug.
>= ; M>  +USING_SYSTEM_COMPILER?=3D no
> M>&nbs= p; +USING_SYSTEM_LINKER?=3D no
> M>  +
> M>  +TEST_SYSTEM_COMPILER_VARS=3D \
>= ; M>  + USING_SYSTEM_COMPILER MK_SYSTEM_COMPILER \
> M>  + MK_CROSS_COMPILER MK_CLANG_BOOTSTRAP \
> M>  + WANT_COMPILER_TYPE WANT_COMPILER_VERSION WANT_COMPIL= ER_VERSION_FILE \
> M>  + WANT_COMPILER_FREEBSD= _VERSION WANT_COMPILER_FREEBSD_VERSION_FILE \
> M>&n= bsp; + CC COMPILER_TYPE COMPILER_FEATURES COMPILER_VERSION \
> M>  + COMPILER_FREEBSD_VERSION \
> M>= ;  + XCC X_COMPILER_TYPE X_COMPILER_FEATURES X_COMPILER_VERSION \
> M>  + X_COMPILER_FREEBSD_VERSION
> M>  +TEST_SYSTEM_LINKER_VARS=3D \
> M>=   + USING_SYSTEM_LINKER MK_SYSTEM_LINKER \
> M>=   + MK_LLD_BOOTSTRAP \
> M>  + WANT_LINKER= _TYPE WANT_LINKER_VERSION WANT_LINKER_VERSION_FILE \
> = M>  + WANT_LINKER_FREEBSD_VERSION WANT_LINKER_FREEBSD_VERSION_FI= LE \
> M>  + LD LINKER_TYPE LINKER_FEATURES LIN= KER_VERSION \
> M>  + LINKER_FREEBSD_VERSION \<= br>
> M>  + XLD X_LINKER_TYPE X_LINKER_FEATURES X_L= INKER_VERSION \
> M>  + X_LINKER_FREEBSD_VERSIO= N
> M>  +
> M>  +.for = _t in compiler linker
> M>  +test-system-${_t}:= .PHONY
> M>  +.for v in ${TEST_SYSTEM_${_t:tu}= _VARS}
> M>  + ${_+_}@printf "%-35s=3D %s\n" "$= {v}" "${${v}}"
> M>  +.endfor
>= ; M>  +.endfor
> M>  +.if (make(buildwo= rld) || make(buildkernel) || make(kernel-toolchain) || \
&= gt; M>  +    make(toolchain) || make(_cross-tools= ))
> M>  +.if ${USING_SYSTEM_COMPILER} =3D=3D "= yes"
> M>  +.info SYSTEM_COMPILER: Determined t= hat CC=3D${CC} matches the source tree.  Not bootstrapping a cross-= compiler.
> M>  +.elif ${MK_CLANG_BOOTSTRAP} =3D= =3D "yes"
> M>  +.info SYSTEM_COMPILER: libclan= g will be built for bootstrapping a cross-compiler.
> M= >  +.endif
> M>  +.if ${USING_SYSTEM_LI= NKER} =3D=3D "yes"
> M>  +.info SYSTEM_LINKER: = Determined that LD=3D${LD} matches the source tree.  Not bootstrapp= ing a cross-linker.
> M>  +.elif ${MK_LLD_BOOTS= TRAP} =3D=3D "yes"
> M>  +.info SYSTEM_LINKER: = libclang will be built for bootstrapping a cross-linker.
&= gt; M>  +.endif
> M>  +.endif
=
> M>  +
> M>  +# End WITH_SYSTE= M_COMPILER / WITH_SYSTEM_LD
> M>  +
> M>  +# Store some compiler metadata for use in installwo= rld where we don't
> M>  +# want to invoke CC a= t all.
> M>  +_TOOLCHAIN_METADATA_VARS=3D COMPI= LER_VERSION \
> M>  + COMPILER_TYPE \
> M>  + COMPILER_FEATURES \
> M&g= t;  + COMPILER_FREEBSD_VERSION \
> M>  = + COMPILER_RESOURCE_DIR \
> M>  + LINKER_= VERSION \
> M>  + LINKER_FEATURES \
> M>  + LINKER_TYPE \
> M> = + LINKER_FREEBSD_VERSION
> M>  +toolchain-m= etadata.mk: .PHONY .META
> M>  + @: > ${.TAR= GET}
> M>  + @echo ".info Using cached toolchai= n metadata from build at $$(hostname) on $$(date)" \
> = M>  +     > ${.TARGET}
> M>= ;  + @echo "_LOADED_TOOLCHAIN_METADATA=3Dt" >> ${.TARGET}
=
> M>  +.for v in ${_TOOLCHAIN_METADATA_VARS}
> M>  + @echo "${v}=3D${${v}}" >> ${.TARGET}
> M>  + @echo "X_${v}=3D${X_${v}}" >> ${.T= ARGET}
> M>  +.endfor
> M>&= nbsp; + @echo ".export ${_TOOLCHAIN_METADATA_VARS}" >> ${.TARGET}<= br>
> M>  + @echo ".export ${_TOOLCHAIN_METADATA_VA= RS:C,^,X_,}" >> ${.TARGET}
> M>  +
> M>  +
> M>  +# We must do = lib/ and libexec/ before bin/ in case of a mid-install error to
> M>  +# keep the users system reasonably usable. = For static->dynamic root upgrades,
> M>  +#= we don't want to install a dynamic binary without rtld and the needed
> M>  +# libraries.  More commonly, for dyn= amic root, we don't want to install a
> M>  +# = binary that requires a newer library version that hasn't been installed = yet.
> M>  +# This ordering is not a guarantee = though.  The only guarantee of a working
> M>&n= bsp; +# system here would require fine-grained ordering of all component= s based
> M>  +# on their dependencies.
> M>  +.if !empty(SUBDIR_OVERRIDE)
> = M>  +SUBDIR=3D ${SUBDIR_OVERRIDE}
> M> = +.else
> M>  +SUBDIR=3D lib libexec
<= div>> M>  +# Add LOCAL_LIB_DIRS, but only if they will not be= picked up as a SUBDIR
> M>  +# of a LOCAL_DIRS= directory.  This allows LOCAL_DIRS=3Dfoo and
> M&= gt;  +# LOCAL_LIB_DIRS=3Dfoo/lib to behave as expected.
> M>  +.for _DIR in ${LOCAL_DIRS:M*/} ${LOCAL_DIRS:N*/:S|$= |/|}
> M>  +_REDUNDANT_LIB_DIRS+=3D  =   ${LOCAL_LIB_DIRS:M${_DIR}*}
> M>  +.endf= or
> M>  +.for _DIR in ${LOCAL_LIB_DIRS}
> M>  +.if ${_DIR} =3D=3D ".WAIT" || (empty(_REDUNDAN= T_LIB_DIRS:M${_DIR}) && exists(${.CURDIR}/${_DIR}/Makefile))
=
> M>  +SUBDIR+=3D ${_DIR}
> M>=   +.endif
> M>  +.endfor
>= ; M>  +.if !defined(NO_ROOT) && (make(installworld) || m= ake(install))
> M>  +# Ensure libraries are ins= talled before progressing.
> M>  +SUBDIR+=3D.WA= IT
> M>  +.endif
> M> = +SUBDIR+=3Dbin
> M>  +.if ${MK_CDDL} !=3D "no"=
> M>  +SUBDIR+=3Dcddl
> M>=   +.endif
> M>  +SUBDIR+=3Dgnu include
=
> M>  +.if ${MK_KERBEROS} !=3D "no"
= > M>  +SUBDIR+=3Dkerberos5
> M>  +.e= ndif
> M>  +.if ${MK_RESCUE} !=3D "no"
> M>  +SUBDIR+=3Drescue
> M>  = +.endif
> M>  +SUBDIR+=3Dsbin
>= ; M>  +.if ${MK_CRYPT} !=3D "no"
> M>  = +SUBDIR+=3Dsecure
> M>  +.endif
&= gt; M>  +.if !defined(NO_SHARE)
> M>  += SUBDIR+=3Dshare
> M>  +.endif
>= ; M>  +.if ${MK_BOOT} !=3D "no"
> M>  += SUBDIR+=3Dstand
> M>  +.endif
>= ; M>  +SUBDIR+=3Dsys usr.bin usr.sbin
> M>&n= bsp; +.if ${MK_TESTS} !=3D "no"
> M>  +SUBDIR+=3D= tests
> M>  +.endif
> M>&n= bsp; +
> M>  +# Local directories are built in = parallel with the base system directories.
> M> = ; +# Users may insert a .WAIT directive at the beginning or elsewhere wi= thin
> M>  +# the LOCAL_DIRS and LOCAL_LIB_DIRS= lists as needed.
> M>  +.for _DIR in ${LOCAL_D= IRS}
> M>  +.if ${_DIR} =3D=3D ".WAIT" || exist= s(${.CURDIR}/${_DIR}/Makefile)
> M>  +SUBDIR+=3D= ${_DIR}
> M>  +.endif
> M>=   +.endfor
> M>  +
> M>= ;  +# We must do etc/ last as it hooks into building the man whatis= file
> M>  +# by calling 'makedb' in share/man= .  This is only relevant for
> M>  +# inst= all/distribute so they build the whatis file after every manpage is
<= /div>
> M>  +# installed.
> M>  = +.if make(installworld) || make(install)
> M>  = +SUBDIR+=3D.WAIT
> M>  +.endif
&g= t; M>  +SUBDIR+=3Detc
> M>  +
=
> M>  +.endif # !empty(SUBDIR_OVERRIDE)
&g= t; M>  +
> M>  +.if defined(NOCLEAN)
> M>  +.warning The src.conf WITHOUT_CLEAN option= can now be used instead of NOCLEAN.
> M>  +MK_= CLEAN:=3D no
> M>  +.endif
> M= >  +.if defined(NO_CLEAN)
> M>  +.info = The src.conf WITHOUT_CLEAN option can now be used instead of NO_CLEAN.
> M>  +MK_CLEAN:=3D no
> M>&= nbsp; +.endif
> M>  +.if defined(NO_CLEANDIR)
> M>  +CLEANDIR=3D clean cleandepend
> M>  +.else
> M>  +CLEANDIR=3D c= leandir
> M>  +.endif
> M>&= nbsp; +
> M>  +.if defined(WORLDFAST)
=
> M>  +MK_CLEAN:=3D no
> M>  +N= O_OBJWALK=3D t
> M>  +.endif
>= M>  +
> M>  +.if ${MK_META_MODE} =3D=3D= "yes"
> M>  +# If filemon is used then we can = rely on the build being incremental-safe.
> M> = +# The .meta files will also track the build command and rebuild should=
> M>  +# it change.
> M>&n= bsp; +.if empty(.MAKE.MODE:Mnofilemon)
> M>  +M= K_CLEAN:=3D no
> M>  +.endif
>= M>  +.endif
> M>  +.if defined(NO_OBJW= ALK) || ${MK_AUTO_OBJ} =3D=3D "yes"
> M>  +NO_O= BJWALK=3D t
> M>  +NO_KERNELOBJ=3D t
<= div>> M>  +.endif
> M>  +.if !define= d(NO_OBJWALK)
> M>  +_obj=3D obj
> M>  +.endif
> M>  +
> M>  +LOCAL_TOOL_DIRS?=3D
> M>  +P= ACKAGEDIR?=3D ${DESTDIR}/${DISTDIR}
> M>  +
=
> M>  +.if empty(SHELL:M*csh*)
> = M>  +BUILDENV_SHELL?=3D${SHELL}
> M>  += .else
> M>  +BUILDENV_SHELL?=3D/bin/sh
> M>  +.endif
> M>  +
> M>  +.if !defined(_MKSHOWCONFIG)
> M= >  +.if !defined(VCS_REVISION) || empty(VCS_REVISION)
<= div>> M>  +.if !defined(SVNVERSION_CMD) || empty(SVNVERSION_C= MD)
> M>  +. for _D in ${PATH:S,:, ,g}
> M>  +.  if exists(${_D}/svnversion)
> M>  +SVNVERSION_CMD?=3D${_D}/svnversion
>= ; M>  +.  endif
> M>  +.  if = exists(${_D}/svnliteversion)
> M>  +SVNVERSION_= CMD?=3D${_D}/svnliteversion
> M>  +.  endi= f
> M>  +. endfor
> M> = ; +.endif
> M>  +.if defined(SVNVERSION_CMD) &a= mp;& !empty(SVNVERSION_CMD)
> M>  +_VCS_REV= ISION?=3D $$(eval ${SVNVERSION_CMD} ${SRCDIR})
> M>&= nbsp; +. if !empty(_VCS_REVISION)
> M>  +VCS_RE= VISION=3D $$(echo r${_VCS_REVISION})
> M>  +.ex= port VCS_REVISION
> M>  +. endif
= > M>  +.endif
> M>  +.endif
> M>  +
> M>  +.if !defined(GI= T_CMD) || empty(GIT_CMD)
> M>  +. for _P in /us= r/bin /usr/local/bin
> M>  +.  if exists($= {_P}/git)
> M>  +GIT_CMD=3D   ${_P}/g= it
> M>  +.  endif
> M>= ;  +. endfor
> M>  +.export GIT_CMD
> M>  +.endif
> M>  +
> M>  +.if !defined(OSRELDATE)
> M&g= t;  +.if exists(/usr/include/osreldate.h)
> M>&= nbsp; +OSRELDATE!=3D awk '/^\#define[[:space:]]*__FreeBSD_version/ { pri= nt $$3 }' \
> M>  + /usr/include/osreldate.h
> M>  +.else
> M>  +OSR= ELDATE=3D 0
> M>  +.endif
> M&= gt;  +.export OSRELDATE
> M>  +.endif
<= /div>
> M>  +
> M>  +# Set VERSI= ON for CTFMERGE to use via the default CTFFLAGS=3D-L VERSION.
<= div>> M>  +.for _V in BRANCH REVISION TYPE
>= M>  +.if !defined(_${_V})
> M>  +_${_V= }!=3D eval $$(awk '/^${_V}=3D/{print}' ${SRCTOP}/sys/conf/newvers.sh); e= cho $$${_V}
> M>  +.export _${_V}
> M>  +.endif
> M>  +.endfor
> M>  +.if !defined(SRCRELDATE)
> M&= gt;  +SRCRELDATE!=3D awk '/^\#define[[:space:]]*__FreeBSD_version/ = { print $$3 }' \
> M>  + ${SRCDIR}/sys/sys/par= am.h
> M>  +.export SRCRELDATE
&g= t; M>  +.endif
> M>  +.if !defined(VERS= ION)
> M>  +VERSION=3D FreeBSD ${_REVISION}-${_= BRANCH:C/-p[0-9]+$//} ${TARGET_ARCH} ${SRCRELDATE}
> M&= gt;  +.export VERSION
> M>  +.endif
> M>  +MAJOR_REVISION=3D ${_REVISION:R}
> M>  +
> M>  +.if !defined(PKG_VER= SION)
> M>  +_STRTIMENOW=3D %Y%m%d%H%M%S
> M>  +_TIMENOW=3D ${_STRTIMENOW:gmtime}
> M>  +.if ${_BRANCH:MCURRENT*} || ${_BRANCH:MSTABLE*} || $= {_BRANCH:MPRERELEASE*}
> M>  +EXTRA_REVISION=3D= .snap${_TIMENOW}
> M>  +.elif ${_BRANCH:MALPHA= *}
> M>  +EXTRA_REVISION=3D .a${_BRANCH:C/ALPHA= ([0-9]+).*/\1/}.${_TIMENOW}
> M>  +.elif ${_BRA= NCH:MBETA*}
> M>  +EXTRA_REVISION=3D .b${_BRANC= H:C/BETA([0-9]+).*/\1/}.${_TIMENOW}
> M>  +.eli= f ${_BRANCH:MRC*}
> M>  +EXTRA_REVISION=3D .rc$= {_BRANCH:C/RC([0-9]+).*/\1/}.${_TIMENOW}
> M>  = +.elif ${_BRANCH:M*-p*}
> M>  +EXTRA_REVISION=3D= p${_BRANCH:C/.*-p([0-9]+$)/\1/}
> M>  +.endif<= br>
> M>  +PKG_VERSION:=3D ${MAJOR_REVISION}${EXTRA= _REVISION:C/[[:space:]]//g}
> M>  +.endif
> M>  +.endif # !defined(PKG_VERSION)
= > M>  +
> M>  +.if !defined(PKG_TIME= STAMP)
> M>  +TIMEEPOCHNOW=3D %s
> M>  +SOURCE_DATE_EPOCH=3D ${TIMEEPOCHNOW:gmtime}
=
> M>  +.else
> M>  +SOURCE_DATE= _EPOCH=3D ${PKG_TIMESTAMP}
> M>  +.endif
> M>  +
> M>  +PKG_NAME_PREF= IX?=3D FreeBSD
> M>  +PKG_MAINTAINER?=3D re@FreeBSD.org
> M>&= nbsp; +PKG_WWW?=3D https://www.FreeB= SD.org
> M>  +.export PKG_NAME_PREFIX
> M>  +.export PKG_MAINTAINER
> M&g= t;  +.export PKG_WWW
> M>  +
> M>  +.if !defined(_MKSHOWCONFIG)
> M>&= nbsp; +_CPUTYPE!=3D MAKEFLAGS=3D CPUTYPE=3D${_TARGET_CPUTYPE} ${MAKE} -f= /dev/null \
> M>  + -m ${.CURDIR}/share/mk MK= _AUTO_OBJ=3Dno -V CPUTYPE
> M>  +.if ${_CPUTYPE= } !=3D ${_TARGET_CPUTYPE}
> M>  +.error CPUTYPE= global should be set with ?=3D.
> M>  +.endif<= br>
> M>  +.endif
> M>  +.= if make(buildworld)
> M>  +BUILD_ARCH!=3D uname= -p
> M>  +# On some Linux systems uname -p ret= urns "unknown" so skip this check there.
> M>  = +# This check only exists to tell people to use TARGET_ARCH instead of
> M>  +# MACHINE_ARCH so skipping it when crossb= uilding on non-FreeBSD should be fine.
> M>  +.= if ${MACHINE_ARCH} !=3D ${BUILD_ARCH} && ${.MAKE.OS} =3D=3D "Fre= eBSD"
> M>  +.error To cross-build, set TARGET_= ARCH.
> M>  +.endif
> M>&nb= sp; +.endif
> M>  +WORLDTMP?=3D ${OBJTOP}/tmp
> M>  +BPATH=3D ${CCACHE_WRAPPER_PATH_PFX}${WOR= LDTMP}/legacy/usr/sbin:${WORLDTMP}/legacy/usr/bin:${WORLDTMP}/legacy/bin= :${WORLDTMP}/legacy/usr/libexec
> M>  +XPATH=3D= ${WORLDTMP}/bin:${WORLDTMP}/usr/sbin:${WORLDTMP}/usr/bin
> M>  +
> M>  +# When building we w= ant to find the cross tools before the host tools in ${BPATH}.
=
> M>  +# We also need to add UNIVERSE_TOOLCHAIN_PATH so = that we can find the shared
> M>  +# toolchain = files (clang, lld, etc.) during make universe/tinderbox
&g= t; M>  +STRICTTMPPATH=3D ${XPATH}:${BPATH}:${UNIVERSE_TOOLCHAIN_= PATH}
> M>  +# We should not be using tools fro= m /usr/bin accidentally since this could cause
> M>&= nbsp; +# the build to break on other systems that don't have that tool. = For now we
> M>  +# still allow using the old b= ehaviour (inheriting $PATH) if
> M>  +# BUILD_W= ITH_STRICT_TMPPATH is set to 0 but this will eventually be removed.
<= /div>
> M>  +
> M>  +# Currently= strict $PATH can cause build failures. Once the remaining issues
> M>  +# have been resolved it will be turned on by d= efault.
> M>  +BUILD_WITH_STRICT_TMPPATH?=3D0
> M>  +.if defined(CROSSBUILD_HOST)
> M>  +# When building on non-FreeBSD we can't rely on the = tools in /usr/bin being compatible
> M>  +# wit= h what FreeBSD expects. Therefore we only use tools from STRICTTMPPATH
> M>  +# during the world build stage. We build = most tools during the bootstrap-tools
> M>  +# = phase but symlink host tools that are known to work instead of building = them
> M>  +BUILD_WITH_STRICT_TMPPATH:=3D1
<= /div>
> M>  +.endif
> M>  +.if $= {BUILD_WITH_STRICT_TMPPATH} !=3D 0
> M>  +TMPPA= TH=3D ${STRICTTMPPATH}
> M>  +.else
> M>  +TMPPATH=3D ${STRICTTMPPATH}:${PATH}
&= gt; M>  +.endif
> M>  +
&= gt; M>  +#
> M>  +# Avoid running mktem= p(1) unless actually needed.
> M>  +# It may no= t be functional, e.g., due to new ABI
> M>  +# = when in the middle of installing over this system.
> M&= gt;  +#
> M>  +.if make(distributeworld) |= | make(installworld) || make(stageworld)
> M>  = +.if ${BUILD_WITH_STRICT_TMPPATH} !=3D 0
> M>  = +MKTEMP=3D${WORLDTMP}/legacy/usr/bin/mktemp
> M>&nbs= p; +.if !exists(${MKTEMP})
> M>  +.error mktemp= binary doesn't exist in expected location: ${MKTEMP}
>= M>  +.endif
> M>  +.else
> M>  +MKTEMP=3Dmktemp
> M>  +.endi= f
> M>  +INSTALLTMP!=3D ${MKTEMP} -d -u -t inst= all
> M>  +
> M>  +.if= ${.MAKE.OS} =3D=3D "FreeBSD"
> M>  +# When bui= lding on FreeBSD we always copy the host tools instead of linking
> M>  +# into INSTALLTMP to avoid issues with incompa= tible libraries (see r364030).
> M>  +# Note: w= e could create links if we don't intend to update the current machine.
> M>  +INSTALLTMP_COPY_HOST_TOOL=3Dcp
<= div>> M>  +.else
> M>  +# However, t= his is not necessary on Linux/macOS. Additionally, copying the host
<= /div>
> M>  +# tools to another directory with cp results= in AMFI Launch Constraint
> M>  +# Violations = on macOS Ventura as part of its System Integrity Protection.
> M>  +INSTALLTMP_COPY_HOST_TOOL=3Dln -s
>= ; M>  +.endif
> M>  +.endif
> M>  +
> M>  +.if make(stagekern= el) || make(distributekernel)
> M>  +TAGS+=3D = kernel
> M>  +PACKAGE=3D kernel
&= gt; M>  +.endif
> M>  +
&= gt; M>  +#
> M>  +# Building a world go= es through the following stages
> M>  +#
> M>  +# 1. legacy stage [BMAKE]
> M= >  +# This stage is responsible for creating compatibility
> M>  +# shims that are needed by the bootstrap-tool= s,
> M>  +# build-tools and cross-tools stages.= These are generally
> M>  +# APIs that tools f= rom one of those three stages need to
> M>  +# = build that aren't present on the host.
> M>  +#= 1. bootstrap-tools stage [BMAKE]
> M>  +# This= stage is responsible for creating programs that
> M>= ;  +# are needed for backward compatibility reasons. They
=
> M>  +# are not built as cross-tools.
>= ; M>  +# 2. build-tools stage [TMAKE]
> M>&n= bsp; +# This stage is responsible for creating the object
= > M>  +# tree and building any tools that are needed during
> M>  +# the build process. Some programs are li= sted during
> M>  +# this phase because they bu= ild binaries to generate
> M>  +# files needed = to build these programs. This stage also
> M>  = +# builds the 'build-tools' target rather than 'all'.
>= M>  +# 3. cross-tools stage [XMAKE]
> M>&nb= sp; +# This stage is responsible for creating any tools that
> M>  +# are needed for building the system. A cross-compi= ler is one
> M>  +# of them. This differs from = build tools in two ways:
> M>  +# 1. the 'all' = target is built rather than 'build-tools'
> M> = +# 2. these tools are installed into TMPPATH for stage 4.
> M>  +# 4. world stage [WMAKE]
> M>&nbs= p; +# This stage actually builds the world.
> M>&nbs= p; +# 5. install stage (optional) [IMAKE]
> M> = +# This stage installs a previously built world.
> M&g= t;  +#
> M>  +
> M>&nb= sp; +BOOTSTRAPPING?=3D 0
> M>  +# Keep these in= sync
> M>  +MINIMUM_SUPPORTED_OSREL?=3D 110400= 1
> M>  +MINIMUM_SUPPORTED_REL?=3D 11.4
> M>  +
> M>  +# Common envir= onment for world related stages
> M>  +CROSSENV= +=3D \
> M>  + MACHINE_ARCH=3D${TARGET_ARCH} \=
> M>  + MACHINE=3D${TARGET} \
&= gt; M>  + CPUTYPE=3D${TARGET_CPUTYPE}
> M>&= nbsp; +.if ${MK_META_MODE} !=3D "no"
> M>  +# D= on't rebuild build-tools targets during normal build.
>= M>  +CROSSENV+=3D BUILD_TOOLS_META=3D.NOMETA
>= M>  +.endif
> M>  +.if defined(TARGET_= CFLAGS)
> M>  +CROSSENV+=3D ${TARGET_CFLAGS}
> M>  +.endif
> M>  +.if= (${TARGET} !=3D ${MACHINE} && !defined(WITH_LOCAL_MODULES)) || = \
> M>  +    defined(WITHOUT_LOC= AL_MODULES)
> M>  +CROSSENV+=3D LOCAL_MODULES=3D=
> M>  +.endif
> M>  +=
> M>  +BOOTSTRAPPING_OSRELDATE?=3D${OSRELDATE}=
> M>  +
> M>  +# boot= strap-tools stage
> M>  +BMAKEENV=3D INSTALL=3D= "sh ${.CURDIR}/tools/install.sh" \
> M>  + TOO= LS_PREFIX=3D${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \
> M>= ;  + PATH=3D${BPATH:Q}:${PATH:Q} \
> M>  = + WORLDTMP=3D${WORLDTMP} \
> M>  + MAKEFLAGS=3D= "-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}"
> M>&nb= sp; +# need to keep this in sync with targets/pseudo/bootstrap-tools/Mak= efile
> M>  +BSARGS=3D DESTDIR=3D \
<= div>> M>  + OBJTOP=3D'${WORLDTMP}/obj-tools' \
> M>  + OBJROOT=3D'$${OBJTOP}/' \
> M>&= nbsp; + UNIVERSE_TOOLCHAIN_PATH=3D${UNIVERSE_TOOLCHAIN_PATH} \
> M>  + MAKEOBJDIRPREFIX=3D \
> M>=   + BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \
>= ; M>  + BWPHASE=3D${.TARGET:C,^_,,} \
> M>&= nbsp; + -DNO_CPU_CFLAGS \
> M>  + -DNO_LINT \=
> M>  + -DNO_PIC \
> M>&n= bsp; + -DNO_SHARED \
> M>  + MK_ASAN=3Dno \
> M>  + MK_CTF=3Dno \
> M>&= nbsp; + MK_CLANG_EXTRAS=3Dno \
> M>  + MK_CLA= NG_FORMAT=3Dno \
> M>  + MK_CLANG_FULL=3Dno \<= br>
> M>  + MK_HTML=3Dno \
> M>= ;  + MK_MAN=3Dno \
> M>  + MK_PROFILE=3D= no \
> M>  + MK_RETPOLINE=3Dno \
> M>  + MK_SSP=3Dno \
> M>  + MK_= TESTS=3Dno \
> M>  + MK_UBSAN=3Dno \
=
> M>  + MK_WERROR=3Dno \
> M> = + MK_INCLUDES=3Dyes \
> M>  + MK_MAN_UTILS=3D= yes
> M>  +
> M>  +BMA= KE=3D \
> M>  + ${TIME_ENV} ${BMAKEENV} ${MAK= E} ${WORLD_FLAGS} -f Makefile.inc1 \
> M>  + $= {BSARGS}
> M>  +.if empty(.MAKEOVERRIDES:MMK_LL= VM_TARGET_ALL)
> M>  +BMAKE+=3D MK_LLVM_TARGET= _ALL=3Dno
> M>  +.endif
> M>= ;  +
> M>  +# build-tools stage
<= div>> M>  +TMAKE=3D \
> M>  + ${TI= ME_ENV} ${BMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc1 \
<= div>> M>  + TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} = \
> M>  + DESTDIR=3D \
> M>= ;  + BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \
&g= t; M>  + BWPHASE=3D${.TARGET:C,^_,,} \
> M>=   + -DNO_CPU_CFLAGS \
> M>  + -DNO_LINT = \
> M>  + MK_ASAN=3Dno \
> M&= gt;  + MK_CTF=3Dno \
> M>  + MK_CLANG_EX= TRAS=3Dno \
> M>  + MK_CLANG_FORMAT=3Dno \
=
> M>  + MK_CLANG_FULL=3Dno \
> M= >  + MK_LLDB=3Dno \
> M>  + MK_RETPOL= INE=3Dno \
> M>  + MK_SSP=3Dno \
> M>  + MK_TESTS=3Dno \
> M>  + M= K_UBSAN=3Dno \
> M>  + MK_WERROR=3Dno
> M>  +
> M>  +# cross-tools s= tage
> M>  +# TOOLS_PREFIX set in BMAKE
> M>  +XMAKE=3D ${BMAKE} \
> M>&n= bsp; + TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} \
= > M>  + MK_LLDB=3Dno \
> M>  + MK_= LLVM_BINUTILS=3Dno \
> M>  + MK_TESTS=3Dno
=
> M>  +
> M>  +# kernel-t= ools stage
> M>  +KTMAKEENV=3D INSTALL=3D"sh ${= .CURDIR}/tools/install.sh" \
> M>  + PATH=3D${= BPATH:Q}:${PATH:Q} \
> M>  + WORLDTMP=3D${WORL= DTMP} \
> M>  + MAKEFLAGS=3D"-m ${.CURDIR}/too= ls/build/mk ${.MAKEFLAGS}"
> M>  +
> M>  +KTMAKE=3D ${TIME_ENV} \
> M>&nb= sp; + TOOLS_PREFIX=3D${TOOLS_PREFIX_UNDEF:U${WORLDTMP}} \
> M>  + ${KTMAKEENV} ${MAKE} ${WORLD_FLAGS} -f Makefile.inc= 1 \
> M>  + DESTDIR=3D \
> M&= gt;  + OBJTOP=3D'${WORLDTMP}/obj-kernel-tools' \
>= ; M>  + OBJROOT=3D'$${OBJTOP}/' \
> M> = ; + UNIVERSE_TOOLCHAIN_PATH=3D${UNIVERSE_TOOLCHAIN_PATH} \
> M>  + MAKEOBJDIRPREFIX=3D \
> M>&nbs= p; + BOOTSTRAPPING=3D${BOOTSTRAPPING_OSRELDATE} \
> M&= gt;  + -DNO_CPU_CFLAGS \
> M>  + -DNO_LI= NT \
> M>  + -DNO_PIC \
> M&g= t;  + -DNO_SHARED \
> M>  + MK_CTF=3Dno = \
> M>  + MK_HTML=3Dno \
> M&= gt;  + MK_MAN=3Dno \
> M>  + MK_PROFILE=3D= no \
> M>  + MK_SSP=3Dno \
> = M>  + MK_RETPOLINE=3Dno \
> M>  + MK_= WERROR=3Dno
> M>  +
> M>&nb= sp; +# world stage
> M>  +WMAKEENV=3D ${CROSSEN= V} \
> M>  + INSTALL=3D"${INSTALL_CMD} -U" \
> M>  + PATH=3D${TMPPATH:Q} \
>= ; M>  + SYSROOT=3D${WORLDTMP}
> M>  +<= br>
> M>  +# make hierarchy
> M>= ; *** 5431 LINES SKIPPED ***
> 

=
-- 
Alexander Motin


--06e1cace14e44ba6b615e2ddd8eabb7c--