From nobody Tue Jan 11 00:11:10 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 217F31930B93 for ; Tue, 11 Jan 2022 00:11:13 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXrhH6scNz3MYl for ; Tue, 11 Jan 2022 00:11:11 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x835.google.com with SMTP id l17so16849049qtk.7 for ; Mon, 10 Jan 2022 16:11:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=yP5ZT4O0Gl16LsjKLYpSqOxTdOROxGVKkGdJ77dsglA=; b=Z8q0LuTVKd62SeR5SIT90Vu65i+fEW7+xXIBLvLK/h3Hj7S6yMQRmqOg47I7izEnzO 2g/MaV3DOfmJUsXpmoHXxoCjecpnGjr5ifrEAmUY2L2FWZk0Ddb1aSoqD2sXqW+3CVtP 6mdRQPOYj/ANeB63smUmbBJj2OO6q3aPEiCgbwlj4HLzmHq0vFlccqJ5n+LMOJH8webd hqsnkvY83H5yPu9EKTckDKa0sfKnoQHzOYwzjdkwaXhv/ezTf170kv13mucKMTs3IHjU 1wmC4za4SmFQcGikGLETPNUBLqLH9oCcP2U9+poM6uHimJhY87PEx21KZzWdlDfrzBJU uaEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=yP5ZT4O0Gl16LsjKLYpSqOxTdOROxGVKkGdJ77dsglA=; b=dax0C1VCbJRfW5VRFR/JppeHKZm+NrzYs6qlH+BuLR+KEFvx+lu/M4ndlOQVnD673P MLfB1ScyOg+EizXigfi8TqfhSAIY4k59eQAyibtoVieB7ITikY4NHW3JWCn95nHfP68H FlPF2p6iArIlo1obZlB20Nim0MS/CywXo4Zez454U/h/e8pj5y5fXxKLW/PR2RWkh8Pv JYv8FtAKt//tmUeijYUvxThk4wIo2UX9jYkc+TURI5eJUrI871J1kiJ5kObejecjiVfq QgTyLMKhXviKcnsz9HotMvxV11Dmw+7xNPUnUKT89w0hlqOeG7gy2f+oIstb/e3BN0xF Ry1Q== X-Gm-Message-State: AOAM530KRppWJvvJaLcTBOENfvz/x4pZitsNH5oEDRvj9+gcvXEQvHD6 ODC3uiyLWrZjDBUbeoEqf7Jsqw== X-Google-Smtp-Source: ABdhPJwJspFcC/62T8713fMM8Jj6zyR/y1XMEwwrUVhBpnRgJTzdDlFKkwsMiWaJGCICyWhgELkNow== X-Received: by 2002:a05:622a:178e:: with SMTP id s14mr256590qtk.688.1641859871388; Mon, 10 Jan 2022 16:11:11 -0800 (PST) Received: from mutt-hbsd (pool-100-16-224-136.bltmmd.fios.verizon.net. [100.16.224.136]) by smtp.gmail.com with ESMTPSA id y17sm2855629qtw.1.2022.01.10.16.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 16:11:10 -0800 (PST) Date: Mon, 10 Jan 2022 19:11:10 -0500 From: Shawn Webb To: Mateusz Guzik Cc: Mark Johnston , freebsd-hackers@freebsd.org Subject: Re: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets Message-ID: <20220111001110.medkloif6zghtatg@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <20220110221116.gustgfgfge6pb5fe@mutt-hbsd> <20220111001023.wx5nh64a5zqq7cae@mutt-hbsd> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="vbzw4udfy3tf73no" Content-Disposition: inline In-Reply-To: <20220111001023.wx5nh64a5zqq7cae@mutt-hbsd> X-Rspamd-Queue-Id: 4JXrhH6scNz3MYl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=Z8q0LuTV; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::835 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-4.69 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; NEURAL_HAM_SHORT(-0.59)[-0.587]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; SUBJECT_HAS_QUESTION(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[100.16.224.136:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::835:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --vbzw4udfy3tf73no Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 10, 2022 at 07:10:23PM -0500, Shawn Webb wrote: > On Tue, Jan 11, 2022 at 12:43:06AM +0100, Mateusz Guzik wrote: > > On 1/11/22, Mark Johnston wrote: > > > On Mon, Jan 10, 2022 at 05:11:16PM -0500, Shawn Webb wrote: > > >> Hey all, > > >> > > >> So I'm getting an interesting ZFS-related kernel panic. I've uploaded > > >> the core.txt at [0]. I suspect it's related to FreeBSD commit > > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge > > >> openzfs/zfs@f291fa658 (master) into main). > > >> > > >> I'm able to reproduce it on a single system with some level of > > >> determinism: I'm building the security appliance firmware at ${DAYJO= B} > > >> in a bhyve VM that's backed by a zvol. The host is a Dell Precision > > >> 7540 laptop with a single NVMe drive in it. The VM is configured with > > >> a single zvol, booting with UEFI. > > >> > > >> Looking at the commit email sent to dev-commits-src-all@, I see this: > > >> 146 files changed, 4933 insertions(+), 1572 deletions(-) > > >> > > >> Strangely, when I run `git show > > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset > > >> of those changes. > > > > > > That is a merge commit. You need to specify that you want a diff > > > against the first parent (the preceding FreeBSD), so something > > > equivalent to "git diff --stat 681ce946f^ 681ce946f". Use > > > "git log 681ce946f^2" to see the merged OpenZFS commits. > > > > > >> As a downstream consumer of 14-CURRENT, how am I supposed to even > > >> start debugging such a large patchset in any manner that respects my > > >> time? > > >> > > >> It seems to me that breaking up commits into smaller, bite-size chun= ks > > >> would make life easier for those experiencing bugs, especially ones > > >> that result in kernel panics. > > > > > > That's up to the upstream project, in this case OpenZFS. > > > > > >> ZFS in and of itself is a beast, and I've yet to study any of its > > >> code, so when there's a commit that large, even thinking about > > >> debugging it is a daunting task. > > >> > > >> Needless to say, I'm going to need some hand holding here for > > >> debugging this. Anyone have any idea what's going on? > > > > > > To start, you'll need to look at the stack trace for the thread with = tid > > > 100061. > > > > >=20 > > imo the kernel should be patched to obtain the trace on its own. As > > the target has interrupts disabled it will have to do it with NMI, but > > support for that got scrapped in > >=20 > > commit 1c29da02798d968eb874b86221333a56393a94c3 > > Author: Mark Johnston > > Date: Fri Jan 31 15:43:33 2020 +0000 > >=20 > > Reimplement stack capture of running threads on i386 and amd64. >=20 > I guess it's especially problematic for laptop systems where dropping > to the db> prompt isn't an option (nvidia driver on this laptop). I'd > have to scrap the entire notion of a GUI, which kinda defeats the > purpose of using a laptop. >=20 > Plugging in a USB memstick and setting debug.trace_on_panic=3D0 is the > route I usually take on such systems. Sorry, wrong sysctl node. I meant to reference debug.debugger_on_panic. --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --vbzw4udfy3tf73no Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmHcyx4ACgkQ/y5nonf4 4fpWrw/+IdLPNu9IUlQaq/iXCbhIP4+oCYA/k3KyhjDW6HL77Il2G4dwMbJ+UVGU QEhBAmPi8Ba3JKzxxy6GQwMff2gkC2ieWCltb3VEdzUvx38inyxzjxRCJ76oljTF CpoFvnRKGsl4yGOwJnhgtCogH1fkB6r2N5CTQ23axTYqops6bAt0SAPkdcXMih8B furpjFJElYBm8VUp1RQ53DbTcTALH6wxWqFYjX/yvlAcxQAPJLPmBy2H7WNXN6uX /ynpTleQW5xzY0emL2aB7J2dnw6fCzqC4Rorj6Qw0ipw8fd+3/JNdQbpPlIWwBlp RSdBbsi2MJvay+jFkX3IEW/eKZWBHHwHGHo1sbgDCc4FKWft6PyPooraVMMjMjhN BhxM8XyRWCMgdXbQHq5K4WLvqdZxmcPBTalBshPmLPNnL5CcifPqVQrH6xaO/JNP 0IiV6ilbUXJIDb5sIeT7tSiGyV0Jo8Bhoh+lVc6LPygqHYW6ujcayELTpTY/JrNJ q/AikJ2TiYLV+qBa1VfflGF2y2T0v0itEapN1QzNuA6HKbvABOljd/v5wR6hkHkG rRQHA2UtjVgVfnqiSo5wT1vTD7eagy/rfgwNUQ92OGJcFXRdlT6mTP3CUbRTH3pz zT+wXcjISqZzdkTHcMOZu2O8isLfPRcQY9T5vyIg3Iz/QEBh3eM= =274Z -----END PGP SIGNATURE----- --vbzw4udfy3tf73no--