From nobody Thu Apr 13 06:11:08 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pxq3021p2z45XGN for ; Thu, 13 Apr 2023 06:11:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.gq1.yahoo.com (sonic309-22.consmr.mail.gq1.yahoo.com [98.137.65.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pxq2z1MXDz4FBC for ; Thu, 13 Apr 2023 06:11:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Rb4veZNA; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1681366281; bh=uhYHB/QKC6y9MqB2is1zGkHhHywh5oRKko+KkQ7uWEk=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=Rb4veZNA4DGBh7WVDfIEgq/oDF60pBz6S5+n8hGibfZ1Vb/du3EXTjB1xkFOua90ENCvl1y4iff0ewYHikaa5MDZO5bcfTgP9w645JyZ4Cax7MCChxPs90CuL/rFTFwD0qTe1NMMqr5wrZ3xoxZGgHIyE1I+J/HhEMGCTbI8vXsEMfF24Ai57leYZ6KCak7TNqGN9y2leTMma9qPiHZG+zPLlett7Y4o7MSE6dVgghJ6cK/6CoqILaNdDld57p4pU/BQpfAhe7WxlmvNx34LXyNgpIsoJv9MmYzqZXydSrJPkz5viyinSnfRLuaOs2eK0QoWQu94KuP3j1Mga2iq2A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1681366281; bh=FOZPolu4K3kJUCjyoxoBg8SeqYmxKReXJ0Yh6Ww39yi=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=mbs89pBosxKXrmpqXCo7GHNfq7ZIApAFg0lIU8PLMuB3kTPCPetYjwulQO3yJF58MJFUFqCxfBDgZoahT0TMzVjtzOH8oSi4mdtbqjz5xQHQEWtRUe/n2TEvwObIMYq3LsL93mNeHOn7/fB96mZ7f2dd2AZ784QyxypzSFc7/VB/eh8IlLYRsSmm04JxzTkcdppGc/YK/FDgQkIISRdTbffXLo7Eq+zm7EGlRrmDM+E/DHZm9ILZYsaAZiZFPYEfDUnWPC9GyL+bYz22wG2Hqhowoqr9txrcpy7t1hXkbyFXAeybzc43Gltw6lflv9Dd+TNvPwxKGnzqPbCssbV19A== X-YMail-OSG: GKMQ9c4VM1mGvvt77TU_DZrhesYIyXM5SI90qc80bLgUjub8ivvUMtRuDynQY7k nkLXjyMz3XTRm25mrWgneV_.8gMmZY8hrgbz5YPXn1FVMHhc9eUOgI53DUWWo4XGc7aYp4x5ow5N uzevXtBhItAXCrwEH6_z.fSSLmKkHKsOjlFpsJ2JJHFn19dCYJ0ewPLuob2ABFv6DF1Nhg_IH1o. vYwRnTGiKk0oQXdABh4Q57ZWya5cgYQrib1oQ_5N1rZ2vUbFkgOSc_PRA2_a3SxtFfVLKmGCYwlc 9Oi6unYjaZTJSLGqxoUWYN2Tz6AUN1xvQW9DLowbmLt8zJarloOVigO5bLzStQiWHlQU.Xg4X_tD 5v6M5As0w4quTFbKTcCKZRbB69PxC3n1LdwZWDO4WHS8t.ahFHlnQ1gPcar75tCGpyWLEH9wDkym MAOYYE2.xOBTC2VoYS2bCvQ5Wy2moxv3lsUh0RBA2HVhjH3U2ZeVLJgGPmojwViUSt73jjyfF8OA ZFhF2605PvKpECT4Ll4u6u.SdxXNEf8dFfE1kCjfqyjypFgC1F_6.nAJxa1Q7x2R7Mvx.uRbygoN uzzlvQ1JPY8RSqcWM7x4gB1AjDMZPxP8NkWYBxg1nisVt5PEary3sqJc0u7022TfCJKkm4lGVqxH mxzrHlUTsT30ibBs8OTxbAd92uNyfK39n2T7pDoVKUVMdAyqy.JENDeQe8DGKw95.Ca_Je22HkNb BQvDIlxZnwAmfQTzPLqoMaPSRwziYy1PZodVxtFTmezynFT3v1tW0KyPTQZ_y128seuHUeaCmEFG OdK3FnnoAh_LvheyLKN0FmBBMS3Jmyjx7KoNKBUbUBAJr5AOdcDtDHzgXez2mJvp74nKkvJ5ubw4 y76o5RdpVZ5ep_FqAIOYxkWS8.5soP8kCTzpzOlL4GxYNk9UKghl8huzyBQ2fIU5cWECMcXygdFT cysBtArARhFQdSG_tjevL2J0yH2ftdf4eWpo9WDgbDGCgrdmDwazoMt_GskOp_cY5kDQDsukLKnC tENtgGBQ4hY8Hn1TpjK4jbAw_lJkJOGMvm5N3ahaHmcxDWbm8F6_SfmSDv7czc6QrzI0X7sbTPB6 hLi23YdLgEptz.1H.EmOylyQ5RQz_1V6ISufN5F5F9us4ZNsb9WEfj8u5S_gW0omB.5xxHuHXrvK 4ng3YVtpO8DgYiWU1n2FZ78aXKlTHy_H25SIxiHOAOqeVmViSmDAsgS3r4hHJ1eEdgjSUYQ3yNa_ 7Mw7b5Z455rYlGJQRSfTL4_VduulY.pJz6n8ItVumD.EcZZg5kRQ25vws3YMpdz63AdyNvqPWxMt PjHy5_HWfBdaRcPOykC60DvgGV5sO6T1ktsoCwdK5ZhMuIDGQNPUhAs8.RkQRwHTkQRNZuMokz2t pWkP_1NPt7KzimeSF.154O_.NI9AsRKNScVdiF6hOY2S9fKruMCTyY78zqnqtvvvt29cdN1XcN1p G2AJdAWSKjfXfCZDPwT.gH.LpOweH_fvuuX1bpOqzDukzdb9tkcTUVRz8a_wv_PJftjABNAUq90h ZRg9OFbd_J6UwgqlptE1Eue2WqxdQkKZPotCyaFYlD2VRqH3Mdd9Wi7Yg.cJD6jscEQ2bO21rfRh 3UZKI9n_c.whJMr6mPSOByHZOIhwgQCBcnroe5CgHPhU_pNfyzixUByeBg8wfcXcm142n0AgoGYB KH7ReA7FahtYce4vzWff8PRfCYHLc3bLEoPUZ8i9YUDqOZpiI.YvINK6j.5N8tWnCFzpKcf3UoPR bNI_FtJeE29zmlIADnHzwPd6WU1mKGo8cOyChctKh1ttRFJDHzK44sEVTOFWEFbXlvVeCaBxjAPF Ja.JuL4QLyPfkpe7QusyQOXMB2O5H3.aAL.wTQ77Eu7GrICJZJ4AWAPikU1rrH7fBVOHrfffeCDX mFgKwCZpaHT4_kP9s1z1qjlcj0Ut5WFL8nd1aGB2Nlkw5p7cK44k8xbZFZcauwCWswdU.SpnRxyD r6OycCIQ2PkJtiESJsvcH_tg.d7mKBRJa7Oj.Xxqghe7z4uPGvUVOuYOJsT82eGajhqTj9dqPijz DZodhJiZwf7LxkfHxlJugpVa7Z6uks78arpE7DLz69jAaKhE3Mu8TKx8ORHbyiYBCo4XeDKG0FxT JRFeRhVFpcU3Dj9QY_LGNYTrkdFnCjzG593HqJS28517CczYyurL2sAW.bDKTil.oQtnSM6eJ56f 54_iHVhZP91t4lvVaPsEe1uIFLbEc20_yQwM0DJdmmq2PAfPdeSTenBaP_tGQ8zn_cvhJChJoCFE j1nLd X-Sonic-MF: X-Sonic-ID: b830c80c-4626-4c8f-9a71-a30c5e12f25a Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Thu, 13 Apr 2023 06:11:21 +0000 Received: by hermes--production-ne1-7dbd98dd99-znv4z (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 618c13e3389b1298e3a8008ed2e8b408; Thu, 13 Apr 2023 06:11:20 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 Message-Id: <5C8E3DDB-3D8D-4C7F-94DE-09CF45611E70@yahoo.com> Date: Wed, 12 Apr 2023 23:11:08 -0700 To: Cy Schubert , dev-commits-src-main@freebsd.org X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <5C8E3DDB-3D8D-4C7F-94DE-09CF45611E70.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.148:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[dev-commits-src-main@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.148:from] X-Rspamd-Queue-Id: 4Pxq2z1MXDz4FBC X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N From: Cy Schubert wrote on Date: Thu, 13 Apr 2023 05:47:33 UTC : > On Wed, 12 Apr 2023 22:28:13 -0700 > Mark Millard wrote: >=20 > > From: Charlie Li wrote on > > Date: Wed, 12 Apr 2023 20:11:16 UTC : > >=20 > > > Charlie Li wrote:=20 > > > > Mateusz Guzik wrote:=20 > > > >> can you please test poudriere with > > > >> https://github.com/openzfs/zfs/pull/14739/files > > > >>=20 > > > > After applying, on the md(4)-backed pool regardless of = block_cloning,=20 > > > > the cy@ `cp -R` test reports no differing (ie corrupted) files. = Will=20 > > > > report back on poudriere results (no block_cloning). > > > >=20 > > > As for poudriere, build failures are still rolling in. These are = (and=20 > > > have been) entirely random on every run. Some examples from this = run: > > >=20 > > > lang/php81: > > > - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=20 > > > ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf = ${STAGEDIR}/${PREFIX}/etc > > > - consumers fail to build due to corrupted php.conf packaged > > >=20 > > > devel/ninja: > > > - phase: stage > > > - install -s -m 555=20 > > > /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=20 > > > /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin > > > - consumers fail to build due to corrupted bin/ninja packaged > > >=20 > > > devel/netsurf-buildsystem: > > > - phase: stage > > > - mkdir -p=20 > > > = /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne= tsurf-buildsystem/makefiles=20 > > > = /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne= tsurf-buildsystem/testtools > > > for M in Makefile.top Makefile.tools Makefile.subdir = Makefile.pkgconfig=20 > > > Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do = \ > > > cp makefiles/$M=20 > > > = /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne= tsurf-buildsystem/makefiles/;=20 > > > \ > > > done > > > - graphics/libnsgif fails to build due to NUL characters in=20 > > > Makefile.{clang,subdir}, causing nothing to link=20 > >=20 > > Summary: I have problems building ports into packages > > via poudriere-devel use despite being fully updated/patched > > (as of when I started the experiment), never having enabled > > block_cloning ( still using openzfs-2.1-freebsd ). > >=20 > > In other words, I can confirm other reports that have > > been made. > >=20 > > The details follow. > >=20 > >=20 > > [Written as I was working on setting up for the experiments > > and then executing those experiments, adjusting as I went > > along.] > >=20 > > I've run my own tests in a context that has never had the > > zpool upgrade and that jump from before the openzfs import to > > after the existing commits for trying to fix openzfs on > > FreeBSD. I report on the sequence of activities getting to > > the point of testing as well. > >=20 > > By personal policy I keep my (non-temporary) pool's compatible > > with what the most recent ??.?-RELEASE supports, using > > openzfs-2.1-freebsd for now. The pools involved below have > > never had a zpool upgrade from where they started. (I've no > > pools that have ever had a zpool upgrade.) > >=20 > > (Temporary pools are rare for me, such as this investigation. > > But I'm not testing block_cloning or anything new this time.) > >=20 > > I'll note that I use zfs for bectl, not for redundancy. So > > my evidence is more limited in that respect. > >=20 > > The activities were done on a HoneyComb (16 Cortex-A72 cores). > > The system has and supports ECC RAM, 64 GiBytes of RAM are > > present. > >=20 > > I started by duplicating my normal zfs environment to an > > external USB3 NVMe drive and adjusting the host name and such > > to produce the below. (Non-debug, although I do not strip > > symbols.) : > >=20 > > # uname -apKU > > FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 = main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082 > >=20 > > I then did: git fetch, stash push ., merge --ff-only, stash apply . = : > > my normal procedure. I then also applied the patch from: > >=20 > > https://github.com/openzfs/zfs/pull/14739/files > >=20 > > Then I did: buildworld buildkernel, install them, and rebooted. > >=20 > > The result was: > >=20 > > # uname -apKU > > FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #91 = main-n262122-2ef2c26f3f13-dirty: Wed Apr 12 19:23:35 PDT 2023 = root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400086 1400086 > >=20 > > The later poudriere-devel based build of packages from ports is > > based on: > >=20 > > # ~/fbsd-based-on-what-commit.sh -C /usr/ports > > 4e94ac9eb97f (HEAD -> main, freebsd/main, freebsd/HEAD) = devel/freebsd-gcc12: Bump to 12.2.0. > > Author: John Baldwin > > Commit: John Baldwin > > CommitDate: 2023-03-25 00:06:40 +0000 > > branch: main > > merge-base: 4e94ac9eb97fab16510b74ebcaa9316613182a72 > > merge-base: CommitDate: 2023-03-25 00:06:40 +0000 > > n613214 (--first-parent --count for merge-base) > >=20 > > poudriere attempted to build 476 packages, starting > > with pkg (in order to build the 56 that I explicitly > > indicate that I want). It is my normal set of ports. > > The form of building is biased to allowing a high > > load average compared to the number of hardware > > threads (same as cores here): each builder is allowed > > to use the full count of hardware threads. The build > > used USE_TMPFS=3D"data" instead of the USE_TMPFS=3Dall I > > normally use on the build machine involved. > >=20 > > And it produced some random errors during the attempted > > builds. A type of example that is easy to interpret > > without further exploration is: > >=20 > > pkg_resources.extern.packaging.requirements.InvalidRequirement: = Parse error at "'\x00\x00\x00\x00\x00\x00\x00\x00'": Expected = W:(0-9A-Za-z) > >=20 > > A fair number of errors are of the form: the build > > installing a previously built package for use in the > > builder but later the builder can not find some file > > from the package's installation. > >=20 > > Another error reported was: > >=20 > > ld: error: /usr/local/lib/libblkid.a: unknown file type > >=20 > > For reference: > >=20 > > [main-CA72-bulk_a-default] [2023-04-12_20h45m32s] [committing:] = Queued: 476 Built: 252 Failed: 11 Skipped: 213 Ignored: 0 Fetched: 0 = Tobuild: 0 Time: 00:37:52 > >=20 > > I started another build that tried to build 224 packeges: > > the 11 failed and 213 skipped. > >=20 > > Just 1 package built that failed before: > >=20 > > [00:04:58] [09] [00:04:15] Finished databases/sqlite3@default | = sqlite3-3.41.0_1,1: Success > >=20 > > It seems to be the only one where the original failure was not > > an example of complaining about the missing/corrupted content > > of a package install used for building. So it is an example > > of randomly varying behavior. > >=20 > > That, in turn, allowed: > >=20 > > [00:04:58] [01] [00:00:00] Building security/nss | nss-3.89 > >=20 > > to build but everything else failed or was skipped. > >=20 > > The sqlite3 vs. other failure difference suggests that writes > > have random problems but later reads reliably see the problem > > that resulted (before the content is deleted). > >=20 > >=20 > > After the above: > >=20 > > # zpool status > > pool: zroot > > state: ONLINE > > config: > >=20 > > NAME STATE READ WRITE CKSUM > > zroot ONLINE 0 0 0 > > da0p8 ONLINE 0 0 0 > >=20 > > errors: No known data errors > >=20 > > # zpool scrub zroot > > # zpool status > > pool: zroot > > state: ONLINE > > scan: scrub repaired 0B in 00:16:25 with 0 errors on Wed Apr 12 = 22:15:39 2023 > > config: > >=20 > > NAME STATE READ WRITE CKSUM > > zroot ONLINE 0 0 0 > > da0p8 ONLINE 0 0 0 > >=20 > > errors: No known data errors > >=20 > >=20 > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com >=20 > Did your pools suffer the EXDEV problem? The EXDEV also corrupted = files. As I reported, this was a jump from before the import to as things are tonight (here). So: NO, unless the existing code as of tonight still has the EXDEV problem! Prior to this experiment I'd not progressed any media beyond: main-n261544-cee09bda03c8-dirty Wed Mar 15 20:25:49. > I think, without sufficient investigation we risk jumping to > conclusions. I've taken an extremely cautious approach, rolling back > snapshots (as much as possible, i.e. poudriere datasets) when EXDEV > corruption was encountered. Again: nothing between main-n261544-cee09bda03c8-dirty and main-n262122-2ef2c26f3f13-dirty was involved at any stage. > I did not rollback any snapshots in my MH mail directory. Rolling back > snapshots of my MH maildir would result in loss of email. I have to > live with that corruption. Corrupted files in my outgoing sent email > directory remain: >=20 > slippy$ ugrep -cPa '\x00' ~/.Mail/note | grep -c :1=20 > 53 > slippy$=20 >=20 > There are 53 corrupted files in my note log of 9913 emails. Those = files > will never be fixed. They were corrupted by the EXDEV bug. Any new ZFS > or ZFS patches cannot retroactively remove the corruption from those > files. >=20 > But my poudriere files, because the snapshots were rolled back, were > "repaired" by the rolled back snapshots. >=20 > I'm not convinced that there is presently active corruption since > the problem has been fixed. I am convinced that whatever corruption > that was written at the time will remain forever or until those files > are deleted or replaced -- just like my email files written to disk at > the time. My test results and procedure just do not fit your conclusion that things are okay now if block_clonging is completely avoided. =3D=3D=3D Mark Millard marklmi at yahoo.com