From nobody Tue Oct 25 05:40:21 2022 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MxLPd3DXZz4gkXS for ; Tue, 25 Oct 2022 05:40:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MxLPd1Mpxz3nPG for ; Tue, 25 Oct 2022 05:40:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MxLPd0HvnzTrJ for ; Tue, 25 Oct 2022 05:40:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 29P5eKxW089313 for ; Tue, 25 Oct 2022 05:40:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 29P5eK2j089312 for fs@FreeBSD.org; Tue, 25 Oct 2022 05:40:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 266014] panic: corrupted zfs dataset (zfs issue) Date: Tue, 25 Oct 2022 05:40:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dpy@pobox.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666676421; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iT93RZe1QBUu06UhVtQUkSMcNiefJNDC3WBpXwQEX6k=; b=qEuIotb5fb2JHz9dbG3mgEnRIPwyDaju3BL0M4CjUQmpSOvOsIEuAU7BxBzAfokOuLCtFo awoY1j+oqQKr/nU7TSo7b+nDeLQptBs6mgsbFCrOYzbgmDzPQj7WtqYPvO/0tjq1/zgWIP v8okk72E8xxs/6AY/aKYIVIJ6N5nK7IrvjEOHwmhpECiWobkpeZviMg+aYwLnrQCia2NeJ y1ii5gWDgx9refGWzyr+/KWcqZVNO0elZYZh7ZNwHg1dnjqiQ5JpHfKwaIVMF+rLmyFWbK nwEuH3lc8W0rt0vP62WVCBkuLNtjaApDFGmZZxRMiiw2o3856ctqDXrLlW57UA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666676421; a=rsa-sha256; cv=none; b=Kq1TIlOuAh193ohVhV3JYPBarBmQUQoqTEKqi6CsBzCEpyJk7RDGpIwTadRiXWv4NvEruJ emue2b18obxJA5ugRGeBaMPMqlMeaxTS+UvVu4tAV5+NZ+XhlZzNZNziKwIHa/FmqiBYOF A9RKPk95RLSqwn1sGnpdKloFgOWMJQEpKgCqEuR/wOWTV8jy5Ngvkl9m9FnUxlbp05M08H hN/9eXhC7htjuXxiT0OX8O9AXRjVefdijWcCTuf8QKFQqODhhPHoq8FtSGmPadxa3uu+bJ Owm14b0w2db2lM550kGVNxF8u9e/zIQYl6F/FjUz9jAQQcZ+gQo4mUpu93dgdg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D266014 Duncan changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|panic: on long running find |panic: corrupted zfs |(zfs issue) |dataset (zfs issue) --- Comment #5 from Duncan --- I got back to trying to move forward with this issue (re-enabling full EOD runs) and found out where the problem was. In my nextcloud jail, part of the /usr/src file system would cause a panic = if accessed (i.e. running a find over it). I haven't gotten around to locating the exact directory/file. Now the interesting thing is that this dataset is encrypted and would mount when decrypted (using a key from higher up the filesytem hierarchy (typed in password as part of startup)). The panic would only occur on access to par= ts of the filesytem dataset. I tried replicating the dataset (to keep for later diagnosis), but upon mounting, machine would panic, requiring a boot into single user mode and deleting the copied dataset (probably should just modify "canmount"), before booting would complete without a panic. My backups(?) consisted of dataset replication onto other pools (in the same machine and to another (soon to be offsite machine (running truenas)). Whe= n I entered the key and mounting occurred, both other systems would panic. My only solution (I could think of), was to create a new dataset and copy o= ver (using rsync in this case) all the folders except /usr/src. I copied /usr/= src from another jail. I have renamed and kept the original dataset for potential debugging in the future. Moral of the story: Proof that ZFS replication is actually NOT the same as= a backup. The corruption was propagated in a more virulent form (mount =3D= =3D panic) to the replicated dataset. At some time I would appreciate being able to help someone figure out what = has happened to the dataset, and how to stop similar in the future. It has sha= ken my faith a little (in ZFS). --=20 You are receiving this mail because: You are the assignee for the bug.=