From nobody Sat Sep 09 16:32:33 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RjdnJ3WC0z4tFMM for ; Sat, 9 Sep 2023 16:32:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RjdnH4646z4fKl for ; Sat, 9 Sep 2023 16:32:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=UgzQdwsK; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694277169; bh=weTyd1bs9Rvq/AfYBi6SheE/T1juWcs47saskxKNi5A=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=UgzQdwsKr4C2C9zP1tQFW2MwW2VQqAt7+PpF3eyqV37+pVXuq7l+MKqCsRnfajrm07I5JQnzamUSuqzWQkJEuxSRtxUS+Yqd6YypBeEsQvHj1ygSG5BHXNt2rreaNlidBkvaonrn9G57fg6xJZgJwtDSmorT1qkyfSjdC5XY5SYPOWGRZoVBmOjmWo7W5Y275Jn2DPGUvKvvYrInDjGjlqKzdBk5bPIuwqDsnZ6aCuA0vtAzDD99ODqrYCd0zAjEL/QYckwitBsI4TC9T+nCORuC/S1ZvD1UA85tB3likTPG0ccqqTo496Fufzd+hgYKUsK2E3pfBpIK5KxwgjtWiA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694277169; bh=IaNjl5Xyh86UoYs6V/SLmByy5XhBbxcbF0P12XlyEyD=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=lW32LKSFozRint7dznXgQZlZ0VHA6p6hSz/LvCb5X/qe9MngAfv09Dg/ydhTxtVwVV6eVifsvAbzMWqYT0f1E8Vxk2RVBHrd94b/lE+WTiVFaKv5NwJq0XNeDb3D1A7xAsTc25jagr2WV4POptTL1txRg7mGXbXj0gQ0CYAuGPMSHGqYXClqNw2gHv03W3YxwxOU4/hSiYcUGWozCTKet0fFj7zD5ACVmtUdZ2poAk+zzPqYGip3cSc52ylcofal0NR1vl0drs+oIzYOTBEPsTX7ZbloEfn0UKYvb74R9lQoaIsUV1WplXb5S8AtZsj2BU+VTKU02VUda7Se99aRVA== X-YMail-OSG: BguVTfQVM1mUJxxH9Q_s3TcW_raiwcFx13_4kwzgeFoeHvotY1jIut6eAne_2Zm EwGgQEHF7EQc7UBvj2MM.PSc462TISmnLQ4HdQ1ekzmpbjgK_zGCI_1x6egKXs.y2pJG7vY7C5qk SArlWNlFzgt0kg2JF5YwNjcE3RyoFdbNl1ehRAJCQcAt7lKDr2Oypc3X44IQGz3IcMLTWjkU_8EX Kn6zisshRNFCKTgCnFoDkOlkQkQCnc37hH1bWDwg_Q42I9PdvcIVaK6HrzhVqmynwmbcoiRGbPob .zdiYxR9DlNJ3WSQJWn67c051qmWJouvGqNibEEEi8cBM_NJ_eZXShNvmbQieVyAlYHpmnsV85X_ hi48q3x8l7XxJQtNYjkSKt2vwWExMwRtzbQzbW80hTIU7C2Nbke2RWWE1KojAE5xCVsyM9j8jBXR rfCeayyRxIcMmBk1CoHiiKgmvsc.Om1ViD9Lh7SCwz4ddoexBR.TdyI4_v.DlKomFh3mQg34w3hW IaEzs39KQUtEREFFVkjrFVdyaY.osjX_kxjYU7GdYXYRsAGQKQa0J6G.gb9TSIm6cDiObqn0AY.D 81bQLPFTv_Z.6wRpQQYBsRAxzmpBrY8dU2c_3FasnNXcUp0DjOWN2FrhZWVfKexLfv08v1B4bwWL pyfSl.6aVr_qXaqk_nnhJ6wtJrrBWv0sCbfAq9aWIdFvvi_i7ksEwU3uTFHpbZCqvWcPGmbCIURw XhDxc0fe2BAfjd81pHFw1JBoCs93nUzWAV6G3pM9SKxf8RosaQUgSlQ1MUqK26_vTC.fQg4ZVCTo tQ_PXD.JtkutQhinlnbgrj23JYcTSWly5czKW.Us0Ev_YJdMCIP.Nh6RMZYgLtb4RDyLAvuQXzFD 1ahUaoYET9JX1OmJoE_rmWAZpzAaQlSM3zUS.xQDaKZ7GK.bsZ8Nv0UvwqAoPZB0mXzpyoV1A18. 0eeCSj4vwQl0tYDYXRAISsH0Aqxi6pc.MlE1dOkMdNUsx_L4AHXQ08VRVMsbqhc2DwEvgrvdmcO5 z7KX3O2jVf6KbmK2BILxzqieAfg9WKEOxHmnORAy35NKjhi7.qob84Otdpff_wQgVoaEoYG6.Gom JxtOkHzsBPi6Uu3IBWuXAf0qkt2geawYdLL7c1ZvDnLBQRr9khz8MTphOeI19JY6mGlnaJyqQgvb n1bB8mXJXqHiukgHeuExGUtqxXNRSJZhnTKvRS4wjzEhNxmeGv4swtIVDA6h_m0WoaN0v5LmdWug sL9RZVporD7zfcpJPG1WJS1hJhDKkK_SeMab91CXsjt1aaYB7elBHAOdl7Mxck5cb6aSAQf9iMFF MZue4xZ6s4R5SyjDlxfu5iTxPQ0CfeCj_Vp9SFCgAyEowW0eUgbNy1cf_8NrFNOPTUDTfR1zhqGa 5roc22EjKm2d5eyGAR45by5Kf3aGzLgLkfnHLsLUNc_eRnDmG4tt1SzjqH9C5fYK0ozHOTNpbVTn 1WHep7Cf.V.OMkNwV_XhmEiv2FL.4WXixOaLcW0DvPQ_28bnl58yR2eDqzXqXw0GQ.JhMrRBV0B1 MNYFS7qrbza3W83VWMPgz3N1yYvsQ_vkrvXw1hUfAsPcR8iV1DNI4TKoniOA8O4VPtlPZG.l6cWS K6Yjkzhqa2VJXOqQU7FdqpMSkKL4oZBmJf8jMc4qY7pBA7wIpiJaR4pENTI.eBPbq.0mfcxag5ad Z1BnAT35hEsij4xtGiUmk0sj8Sz03OzFbn9SPGDnHvIO8JzSOdnC9ZRHcj9_9YS9OoQt5GgujJeT bGTbe.zaHjI0_063hlWY4d7ZppPHcS0TkcKW0BvNE9WF4OFQgijPdkhr0qsr.sYyO22nOSCFgDb6 UQlB1QL2zaMvNwweswFFg1GEWYDxIPD1N.3f2FIUAczGSzhnGbw2DNipZ1U84iDjLgjYAu0cWbGH fJGmuK_bfpjWfIgGvScq3tLJ6KA0qMeUsOv4VYggrLxtqpVZ7hl4QJapM4gn7qt97y3DvjoFH5F. VXkwnwK4dmQesVvatqSMmrc1XGClFjT.G.dSnOaa20AkkE.qIlFMDoXfe3bVoCBBG3rbuxEIVYS_ cIUFfW8Ji7ot9NhO.zPoZ0zcMHmehOPUwo67LNQSyETH.OeaZxYmYvi09JKRboJpBKtbv01XLRm0 gfHWD2XT5xffFVetspjU7N79qsfVPn4UGu9MnpLJxXSc1Nu5nF9ktP65qHvDYrkSHwQYz87clnVz 7aJq0Rdc7TJk9vbjQfE8mXIkA6w7ADVe59NOR0vxN38AuwTHac41Tnpm36OcKGlDR8wJRzNj3E.u CwTWKyXDSxA-- X-Sonic-MF: X-Sonic-ID: 38ec0d9c-0e85-44c9-974b-f419a2f472c3 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sat, 9 Sep 2023 16:32:49 +0000 Received: by hermes--production-gq1-6b7c87dcf5-sv5pn (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b392d06b9743fbafddba95b9ff9f6f4a; Sat, 09 Sep 2023 16:32:44 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics From: Mark Millard In-Reply-To: <8746A218-F83A-40E7-95F8-5EC1E36411C1@yahoo.com> Date: Sat, 9 Sep 2023 09:32:33 -0700 Cc: Current FreeBSD , FreeBSD-STABLE Mailing List , Pawel Jakub Dawidek Content-Transfer-Encoding: quoted-printable Message-Id: <1B343698-6865-4761-B514-1539AAE291BC@yahoo.com> References: <7CE2CAAF-8BB0-4422-B194-4A6B0A4BC12C@yahoo.com> <08B7E72B-78F1-4ACA-B09D-E8C34BCE2335@yahoo.com> <20230907184823.GC4090@FreeBSD.org> <4f4e2b68-57e0-a475-e2bd-1f2b8844ebfe@FreeBSD.org> <354C5B8C-4216-4171-B8C2-8E827817F8E5@yahoo.com> <8B8B3707-4B37-4621-8124-D6A77CAF6879@yahoo.com> <15df58d3-4603-132f-112e-d10a6d4419bf@FreeBSD.org> <2a25427c-5a61-3f72-4e31-b7666741d38d@FreeBSD.org> <63717d32-f340-1320-3335-85135d1b62bc@FreeBSD.org> <05C47E15-640D-41AD-9C4C-73A1D5041CF4@yahoo.com> <8746A218-F83A-40E7-95F8-5EC1E36411C1@yahoo.com> To: Martin Matuska , Alexander Motin , Glen Barber X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; BLOCKLISTDE_FAIL(0.00)[98.137.69.147:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; TO_DN_ALL(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RjdnH4646z4fKl On Sep 8, 2023, at 21:54, Mark Millard wrote: > On Sep 8, 2023, at 18:19, Mark Millard wrote: >=20 >> On Sep 8, 2023, at 17:03, Mark Millard wrote: >>=20 >>> On Sep 8, 2023, at 15:30, Martin Matuska wrote: >>>=20 >>>> I can confirm that the patch fixes the panic caused by the provided = script on my test systems. >>>> Mark, would it be possible to try poudriere on your system with a = patched kernel? >>>=20 >>> . . . >>>=20 >>> On 9. 9. 2023 0:09, Alexander Motin wrote: >>>> On 08.09.2023 09:52, Martin Matuska wrote: >>>>> . . . >>>>=20 >>>> Thank you, Martin. I was able to reproduce the issue with your = script and found the cause. >>>>=20 >>>> I first though the issue is triggered by the `cp`, but it appeared = to be triggered by `cat`. It also got copy_file_range() support, but = later than `cp`. That is probably why it slipped through testing. This = patch fixes it for me: https://github.com/openzfs/zfs/pull/15251 . >>>>=20 >>>> Mark, could you please try the patch? >>>=20 >>> If all goes well, this will end up reporting that the >>> poudriere bulk -a is still running but has gotten past, >>> say, 320+ port->package builds finished (so: more than >>> double observed so far for the panic context). Later >>> would be a report with a larger figure. A normal run >>> I might let go for 6000+ ports and 10 hr or so. >>>=20 >>> Notes as I go . . . >>>=20 >>> Patch applied, built, and installed to the test media. >>> Also, booted: >>>=20 >>> # uname -apKU >>> FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 = #75 main-n265228-c9315099f69e-dirty: Thu Sep 7 13:28:47 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-dbg-clang/usr/main-src/amd64.amd= 64/sys/GENERIC-DBG amd64 amd64 1500000 1500000 >>>=20 >>> Note that this is with a debug kernel (-dbg- in path and -DBG in >>> the GENERIC* name). Also, the vintage of what it is based on has: >>>=20 >>> git: 969071be938c - main - vfs: copy_file_range() between multiple = mountpoints of the same fs type >>>=20 >>> The usual sort of sequencing previously reported to get to this >>> point. Media update starts with the rewind to the checkpoint in >>> hopes of avoiding oddities from the later failure. >>>=20 >>> . . . : >>>=20 >>> [main-amd64-bulk_a-default] [2023-09-08_16h31m51s] [parallel_build:] = Queued: 34588 Built: 414 Failed: 0 Skipped: 39 Ignored: 335 = Fetched: 0 Tobuild: 33800 Time: 00:30:41 >>>=20 >>>=20 >>> So 414 and and still building. >>>=20 >>> More later. (It may be a while.) >>>=20 >>=20 >> [main-amd64-bulk_a-default] [2023-09-08_16h31m51s] [parallel_build:] = Queued: 34588 Built: 2013 Failed: 2 Skipped: 179 Ignored: 335 = Fetched: 0 Tobuild: 32059 Time: 01:42:47 >>=20 >> and still going. (FYI: The failures are expected.) >>=20 >> After a while I might stop it and start over with a non-debug >> kernel installed instead. >=20 > I did ^C after 2.5 hr (with 2447 built): >=20 > ^C[02:30:05] Error: Signal SIGINT caught, cleaning up and exiting > [main-amd64-bulk_a-default] [2023-09-08_16h31m51s] [sigint:] Queued: = 34588 Built: 2447 Failed: 5 Skipped: 226 Ignored: 335 Fetched: = 0 Tobuild: 31575 Time: 02:29:59 > [02:30:05] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-09-08_1= 6h31m51s > [02:30:05] Cleaning up > [02:38:04] Unmounting file systems > Exiting with status 1 >=20 > I'll switch it over to a non-debug kernel and, probably, world > and setup/run another test. >=20 > . . . (time goes by) . . . >=20 > Hmm. This did not get sent when I wrote the above. FYI, non-debug > test status: >=20 > [main-amd64-bulk_a-default] [2023-09-08_19h51m52s] [parallel_build:] = Queued: 34588 Built: 2547 Failed: 5 Skipped: 239 Ignored: 335 = Fetched: 0 Tobuild: 31462 Time: 01:59:58 >=20 > I may let it run overnight. I finally stopped it at 7473 built (a little over 13 hrs elapsed): ^C[13:08:30] Error: Signal SIGINT caught, cleaning up and exiting [main-amd64-bulk_a-default] [2023-09-08_19h51m52s] [sigint:] Queued: = 34588 Built: 7473 Failed: 23 Skipped: 798 Ignored: 335 Fetched: = 0 Tobuild: 25959 Time: 13:08:26 [13:08:30] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-09-08_1= 9h51m52s [13:08:31] Cleaning up [13:17:10] Unmounting file systems Exiting with status 1 In part that was more evidence for deadlocks at least being fairly rare as well. None of the failed ones looked odd. (A fair portion are because the bulk -a was mostly doing WITH_DEBUG=3D builds. Many upstreams change library names, some other file names, or paths used for debug builds and ports generally do not cover well building the debug builds for such. I've used these runs to extend my list of exceptions that avoid using WITH_DEBUG .) So no evidence of corruptions. (I do not normally do bulk -a builds. The rare bulk -a runs are normally to check that my configuration of a builder machine still works reasonably --beyond building just the few hundred ports that I normally build. So I should be able to build most any combination that I decide to try.) =3D=3D=3D Mark Millard marklmi at yahoo.com