From nobody Thu Aug 31 19:30:25 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcB8l46Cxz4rW1W for ; Thu, 31 Aug 2023 19:30:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcB8k0nqrz4dMB for ; Thu, 31 Aug 2023 19:30:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=X5ZX8s51; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693510243; bh=PYJ5FEJQRPonLQxRGGhvGFsWSbrX/hqCBWRdnCtuPlk=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=X5ZX8s51Bi9WiMUcIgGgswccqLb0KPwL3m3RlOPMV06nf/Pv7H3IC45NWGD/r1Dr+Av898a9TZL7o0HiGHpy8SsagAMGyr0oZjb0+ucCcdarh8mfugLzRAYlMqD5Cmm2cX4CSly2vVmOkyseAI9gDg+mC1kqVxUHHOY9sZWIMtN9CVP0fzZYLwjQCNE4Stbkn3XFNvPDZa7HldcLYzrUp4A7JXg4YWpPnB8PZgWR05nTOkf4aeKsrwom1y9873seVxIYFQLMkUFj0o46NEl+8gtT0NyprDvb31/eQpUUgtIeWA10fiYizCWQeopWmsJD5j2Uio1/NbBGkFysawoYyA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693510243; bh=9fKJbn5lqoEtSyKa5Q5kjdJ+vNdvpx6h32QZyBPZdlu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kP4C3UcKkbG7v1TkyTM5MkTwuGkcyRfBhdrbQs64fjT4M5XBSU9/ox7ar7Pl1Oi32HKP3hV+3XTySMLDIvwHQbr2EHRn30ykOD/O7B+SRiYohEUxhfBhbWl5j+dcHEfnYPG9s+hpygur863nNePDK8ytU7ijJolSphbXzynl4/bx8dWP1tjlkzFHwqyMjBiTuty2OKScRxrxnaKXXw8qVy1qihIjlMEGhdx/e4MJfWmK7PDnWhK1lT2oEqV04Z5iQzLLmhr/ucqmpln+MXOeX/zIUq+8UIYWEOo+wbhonQeOM0NcJg7rg8aJHkZ2xW/yM0MaEG/dKmpoKPKfOu+iqw== X-YMail-OSG: NICRPhYVM1nBBTWgza1PyQgeNjerRyGqpT9gHPH1lqnUR6_P0f8a0Hl6WMZV.us fYl.1AWn.a6My9GcD1iTBVz.nVX8iLDPSF1L_YttX9owk7shCm64cadLJ1.NPqa2h08XjQPIVzkI Dh3hPPJUTkqH1JEoel9OYlL0OM1q3pf2lwJO397FqrMHkg6oEye.WZyJzAehJ1Mqno5H7pjOGD_L T.iUzoc9gfxk44IvWlNnIxSpkcFoowqsNAard_0s8fVR978P1VJp7KfL13fOCA0UD2vHpxWYi4_8 qhZmk4_vhv_EvkKYhdwiETfGL.oWPjGHNiJikCMFAevY20J4RQRXpvUjHL2qb629ppoDU1IO16e6 78MlyRDbptJJIHflPmi4Vt8vteTb9rFW_vLZBz_BFqRNQPsyT8deYhEvii7N1Al2Grs8Bp.lC3Et Dq2LpvdffrIeru99kO7nyJqq8OkUE7Vcuu5Vlcnm9E18vUYxUP9fTYzdaE1Sk9L5l4gxXM_yvzUZ qvmuvHYAb20YtOAmEYPZ0Yb48AbyZXYdYHKLHWKy1UpXvxezKcsKYwyegVSZ0C2wxOjCcn3eI1nC NVyj3fddsIG_Pmtin0ehmqtXI57bGkbygaaXcB3GNGOGiK9g8Jlu2qAwEFCnIhAA68jQ1SZIOWss d47HOeeRZqgksx6uYAMRloA6jpPXBdCjeuexignzQNyPx9.9PCLzYStEMz87WmJctJK8Hnm2tZs2 ZTXHIhdfvP.yMnnSQRY85YSR80317NuJvmwM6pENBbK_0DmBEvF0t8bLOjD1HfKo_LnRPTkGo_I7 VHmtFQBeU5spKNK8Kyy6YlELHm6OvwIJDkzKKS5RezGGyg7Ozrigi7ffhdo2cQivBMUq6V2D5R8K KXrPbfFqj_wKJk2qu6izhkYh4SuxYGk4kYU03Chb3gW4q3_JaBcGrTQ4Q1Bjb5TeIuVxXv.vsynS TsdQgISPvm1h1e15SP3MqjHvLSqTlHqUuSDx8tgqxBdGWck_edHxgeaFPbNmXLzNayDK_SehjWzQ vhVEvwQlRY_9TOnQgDa7l31x20e0_ISevQcn7DW6ZlUSGYBgqjbgr.rD6JTWvfVcrbO6B0Kf3zii _4Y9koqCIkLl5yOyjLEWH82Ek81UeSWVAX0Lz_UosCmo.Svv0_2ugTP3uRpPZSwoaQm19FrGGY.n Y.V1XYHQXdGF8TsaitMzr98eiomENv07jmGiYoHk6rSBqHv3l_wos3Hf9kayIkKR9cMqglVW1zxY y6rh.mb1GMvRqFg._ffMqLm9nmxU3bKIFq9qkSSvixPTaViFbwWpqhtm3AfzquCUgOivreqCFA7_ j8t_T.OAam25TadQdGbinwFc4oNAHKkJewLYhEKamSTF0hvvISL97KZP7sq2nAkrd1zV3697r65P NqwVvY32w_PwF7d1kipoHmuPTKh63cF6lROBxEUZu5Rz205YjMJny.e3cxEg7lQUCpU5nDrTnCZ_ _lwd8u5UMDz4FrAOE8dqKsqM5hh3iYmm9Ix5PXAri0lQPnfnK1PkXCnuSbnRDBWYJSqKOMCzPb6v BIRknsj9PvcqoScKoQlomj7Cv62zFPNtlijcfrj3Q9E3ccRqhrLcRW5tVzRh8SilKd3jiP2QkzlB Aej7ccT_bI.6cErmy4gwJMksjcgnF4ARHYeCzlVF9Bs6EfyMgzBdq8ufH4yvFW0.YWAW6uPhHBRl FIKByjO3w2SpjmeEwUaEtzqin6js1Ee4BiMXjaeR5PTifGWz1dAR0IAwiZmdq8KW8HgRF0pzWsCG B6AO3qtjpEG6MEWiuC5aim0SiBdE57_h1Yhgi_egTUAnL2xRDoDw0QC5_FcAEPQ_dyekIEtL2pF3 Sry.yvgxmKVXwBjB9StvCCWN1IJXoC5ezRyDDg3pkcCaTmZ798P0xW479IlYyaF8fHpOjpPz9ZyB 50J4rKcUxwxIcyeqzsK7Rgw7YuK2pPWBH7elm7pl6tm7nzeL7MTXDsY15zsdyJ7B7kO_Ltovy883 Ug3kaGaKJz4B2P1YjGIZDdQZpbBhqrDX2xWM5HRTAwFep.qn9RdxWEAS13gqlr0vaZE3g641aFKX SMfIvdXlsUXgr8zwc6k3jhIZJOwf6.daykJOpaE0ta7pzik9DbEK70MezeiDH6I32ntvfESpZCz6 tUf3s.PjzNsXpNYMQ5_Kcbnou8rcJpRgd2WE6kqjV64J8CeME.iWzV7F.zeH98It0D7WVSdTM7Po EveJJbjTXsS_ba40ANhUJmLKZG9DKAhi_ALrshb45qlK3MrJUvwYwyzn9clVwoQ8g0CgLHk8Z4zC 9fPVpsZI- X-Sonic-MF: X-Sonic-ID: a7f672f3-20bf-4a31-8163-da77d8d5fde0 Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 19:30:43 +0000 Received: by hermes--production-bf1-865889d799-xc84r (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 08baccca88b40090f0c5b1d80466c4ef; Thu, 31 Aug 2023 19:30:37 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad Message-Id: Date: Thu, 31 Aug 2023 12:30:25 -0700 Cc: Alexander Motin To: Cy Schubert , dev-commits-src-main@freebsd.org X-Mailer: Apple Mail (2.3731.700.6) References: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.83:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[dev-commits-src-main@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.83:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RcB8k0nqrz4dMB Cy Schubert wrote on Date: Thu, 31 Aug 2023 17:53:50 UTC : > In message <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org>, = Alexander=20 > Motin > writes: > > On 31.08.2023 08:45, Drew Gallatin wrote: > > > On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote: > > >> It is the first time I see a panic like this.=C3=82 I'll think = about it > > >> tomorrow.=C3=82 But I'd appreciate any information on what is = your workload > > >> and what are you doing related to ZIL (O_SYNC, fsync(), = sync=3Dalways, > > >> etc) to trigger it?=C3=82 What is your pool configuration? > > >=20 > > > I'm not Gleb, but this was something at $WORK, so I can perhaps = help.=20 > > > I've included the output of zpool status, and all non-default = settings=20 > > > in the zpool.=C3=82 Note that we don't use a ZIL device. > > > > You don't use SLOG device. ZIL is always with you, just embedded in=20= > > this case. > > > > I tried to think about this for couple hours and still can't see how = can=20 > > this happen. zil_sync() should not call zil_free_lwb() unless the = lwb=20 > > is in LWB_STATE_FLUSH_DONE. To get into LWB_STATE_FLUSH_DONE lwb = should=20 > > first delete all lwb_vdev_tree entries in zil_lwb_write_done(). And = no=20 > > new entries should be added during/after zil_lwb_write_done() due to = set=20 > > zio dependencies. > > > > I've made a patch tuning some assertions for this context:=20 > > https://github.com/openzfs/zfs/pull/15227 . If the issue is=20 > > reproducible, could you please apply it and try again? May be it = give=20 > > us any more clues. >=20 > One thing that circumvents my two problems is reducing poudriere bulk = jobs=20 > from 8 to 5 on my 4 core machines. What about the likes of your ALLOW_MAKE_JOBS status or other constraints (such as MAKE_JOBS_NUMBER) on the parallelism internal to each builder? My earlier high load average test that did not reproduce the problem was allowed to use 32 builders, each allowed to use 32 make jobs. This was for 32 hardware threads (ThreadRipper 1950X). The maximum observed for each the 3 load averages were: 349.68, 264.30, 243.16 (not simultaneously). (I use top patched to monitor and report such maximums for what it observes.) In general the 3 reported load averages were each over 100 for a very long time. Since I did not get the problem despite the high sustained load averages but with no "extra" builders involved and you got what you report, my test does support the number of builders being large compared to the number of hardware threads being what matters for repeatability via poudriere. A weakness in that evidence is that my test predates: Sun, 27 Aug 2023 =E2=80=A2 git: 315ee00fa961 - main - zfs: merge = openzfs/zfs@804414aad Martin Matuska and so was for one stage earlier relative main's openzfs updates. It = had: Thu, 10 Aug 2023 . . . =E2=80=A2 git: cd25b0f740f8 - main - zfs: cherry-pick fix from = openzfs Martin Matuska=20 =E2=80=A2 git: 28d2e3b5dedf - main - zfs: cherry-pick fix from = openzfs Martin Matuska and no uncommitted openzfs patches. =3D=3D=3D Mark Millard marklmi at yahoo.com