From nobody Thu Aug 31 17:53:50 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rc80x02hxz4rQmw; Thu, 31 Aug 2023 17:53:53 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rc80w5lywz4VLp; Thu, 31 Aug 2023 17:53:52 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; none Received: from shw-obgw-4001a.ext.cloudfilter.net ([10.228.9.142]) by cmsmtp with ESMTP id bjt5qLN4xLAoIblrcqKxjI; Thu, 31 Aug 2023 17:53:52 +0000 Received: from spqr.komquats.com ([70.66.152.170]) by cmsmtp with ESMTPA id blraqbjS5HFsOblrbqLwkr; Thu, 31 Aug 2023 17:53:52 +0000 X-Authority-Analysis: v=2.4 cv=XZqaca15 c=1 sm=1 tr=0 ts=64f0d3b0 a=y8EK/9tc/U6QY+pUhnbtgQ==:117 a=y8EK/9tc/U6QY+pUhnbtgQ==:17 a=8nJEP1OIZ-IA:10 a=UttIx32zK-AA:10 a=6I5d2MoRAAAA:8 a=NEAV23lmAAAA:8 a=YxBL1-UpAAAA:8 a=EkcXrb_YAAAA:8 a=KOormpjoEEOd_GiNS3AA:9 a=wPNLvfGTeEIA:10 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id A0AEE2D3; Thu, 31 Aug 2023 10:53:50 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 981F1D5; Thu, 31 Aug 2023 10:53:50 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Alexander Motin cc: Drew Gallatin , Gleb Smirnoff , Martin Matuska , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 315ee00fa961 - main - zfs: merge openzfs/zfs@804414aad In-reply-to: <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org> References: <202308270509.37R596B5048298@gitrepo.freebsd.org> <07faf861-9186-47d1-992a-91d483ea4e9c@app.fastmail.com> <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org> Comments: In-reply-to Alexander Motin message dated "Thu, 31 Aug 2023 12:50:19 -0400." List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Date: Thu, 31 Aug 2023 10:53:50 -0700 Message-Id: <20230831175350.981F1D5@slippy.cwsent.com> X-CMAE-Envelope: MS4xfFubN8MIBGH9zTs6kPYfygihXW+qjdVSMmqiiCt5AjbToUmIOz9+Ej1oa6qUwdDCxWR1W0gt4eMun6wpAVpvFTlHj9llcdwLu5VzGYaswignXz9JUPSY N/ph8cskllvlqLj/zIDzKnUJ4TM9blhTPcCxfbVe6iCmPwud1pd07B/ywSPSp4mAz9MMYSWlFG4wA5sht10qF1GK5zKoRvSUF+N2TvGK+gUtMA634ufP0Ygu ItNmbEHGdwUW+Nc4AKKhllCSTW6V0uFZO/N9vkwSPvQZmrsIEY2YoRAK0nWWhWAZpXJMTmnK6sQOM07/0MXdzNlxMvgUyzZxVuSyjahGOnnTxQILfK99QlwJ hrVHioFLqkI1iiVfDKbyPWyxc1r3SEso5zvIEtW7t28EsXHrZEuO1hnZo/Q7rQAKTdsnHjUp X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US] X-Rspamd-Queue-Id: 4Rc80w5lywz4VLp In message <1db726d4-32c9-e1b8-51d6-981aa51b7825@FreeBSD.org>, Alexander Motin writes: > On 31.08.2023 08:45, Drew Gallatin wrote: > > On Wed, Aug 30, 2023, at 8:01 PM, Alexander Motin wrote: > >> It is the first time I see a panic like this.  I'll think about it > >> tomorrow.  But I'd appreciate any information on what is your workload > >> and what are you doing related to ZIL (O_SYNC, fsync(), sync=always, > >> etc) to trigger it?  What is your pool configuration? > > > > I'm not Gleb, but this was something at $WORK, so I can perhaps help. > > I've included the output of zpool status, and all non-default settings > > in the zpool.  Note that we don't use a ZIL device. > > You don't use SLOG device. ZIL is always with you, just embedded in > this case. > > I tried to think about this for couple hours and still can't see how can > this happen. zil_sync() should not call zil_free_lwb() unless the lwb > is in LWB_STATE_FLUSH_DONE. To get into LWB_STATE_FLUSH_DONE lwb should > first delete all lwb_vdev_tree entries in zil_lwb_write_done(). And no > new entries should be added during/after zil_lwb_write_done() due to set > zio dependencies. > > I've made a patch tuning some assertions for this context: > https://github.com/openzfs/zfs/pull/15227 . If the issue is > reproducible, could you please apply it and try again? May be it give > us any more clues. One thing that circumvents my two problems is reducing poudriere bulk jobs from 8 to 5 on my 4 core machines. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0 =LÐ