From nobody Sat Nov 18 22:03:11 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXnpL2r0fz50jPc for ; Sat, 18 Nov 2023 22:03:22 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oa1-x2b.google.com (mail-oa1-x2b.google.com [IPv6:2001:4860:4864:20::2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXnpL18MPz4djQ; Sat, 18 Nov 2023 22:03:22 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2b.google.com with SMTP id 586e51a60fabf-1f5da5df68eso334888fac.2; Sat, 18 Nov 2023 14:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700345001; x=1700949801; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ex9QfF4KngHHDOYCOOBXCsmXGbjOktgKYHNzrV5gT98=; b=QISNpbAvacgVvP0v/XW+IPCe26EP5AyvLV2cZWEBct8J1Xe1C6fg4d9tMHJ9fu7Xqj aOp/XyHSAy0uXMziW3YmB5dbA7QzRUuMx3yFQe1hX1U+RnjFZjo+YSOM/UofjcT8ZMfS +LQ3EGeyAiXWKpHQ5DwYzYElF1kcOzoCSHzeN+D7CBEl6ruL0OI4/LPKxhbFgY1M4v8a YOaQ++NDAtfyWxdPmQwjIjhmPLrRM+jv98IqGZMJxl7+pgj5o/3VPZX6qdaA4rUU8Q6p SBDe7qVa6Mw0tXYAV1TPPfTNkXYbddQW3xEMqmfcdkBPpb4oPhxvEkZSmdeJ/Ig7Z58m FivQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700345001; x=1700949801; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ex9QfF4KngHHDOYCOOBXCsmXGbjOktgKYHNzrV5gT98=; b=G4hHON2ZlUIAZFgw2GibVxKndBY6MEw9jmP4WzKxRTPo4k+ByCicbYe6cnghD99Q65 oRVi7FN7GulgobNKug54//OJyZfaobg5HpoPfpmOP+t01IC7/BuMhgx+KC5R/cMB8MaJ I/p/wi1QKAPhTs6imZw0GIdyfsxJrLtc4njE460V3/iZiCZ1rz5e81HtBYyqZxk5NQFP RKB1Q2JU8CfuVBQ/OtcH906NGz1o7xl9P/6UxGmExjpujtNxUYNk3kLwT/LmAysYGlF8 jVRo0xCxASfwdwgNZDSkTkqOmHORkf/lNFdZ/7ttZ1wdaPwPbHhM7ivDKFd46qdTdv3J 7t1Q== X-Gm-Message-State: AOJu0Yx8wiG7X+Sq2Cgb9c6SprR9N4unAHfbSsiG52lzfjdmn4MwHo+W orB5Ko+HU2EtllZY4PfpuvolWXJXveteWpYc59EM+lk= X-Google-Smtp-Source: AGHT+IEeuWE9DHSNOf/mMjOcAG+39VnYGUrQ5jMnZtP+sqThNfLscf9tn9h07Ew1z20uFMZ+ZmrZLddU5iJzR25HQsM= X-Received: by 2002:a05:6870:460a:b0:1f4:a631:f5fc with SMTP id z10-20020a056870460a00b001f4a631f5fcmr4193296oao.33.1700345000786; Sat, 18 Nov 2023 14:03:20 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <805acb16-ee6f-4d09-b01b-39d9ae3f8c86@FreeBSD.org> In-Reply-To: <805acb16-ee6f-4d09-b01b-39d9ae3f8c86@FreeBSD.org> From: Rick Macklem Date: Sat, 18 Nov 2023 14:03:11 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Ronald Klop Cc: FreeBSD CURRENT , Rick Macklem Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Queue-Id: 4SXnpL18MPz4djQ On Sat, Nov 18, 2023 at 10:47=E2=80=AFAM Ronald Klop w= rote: > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca. > > > On 11/18/23 17:09, Rick Macklem wrote: > > On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels w= rote: > >> > >> On 17 Nov 2023, at 22:14, Mike Karels wrote: > >> > >>> On 17 Nov 2023, at 21:24, Rick Macklem wrote: > >>> > >>>> Most of the changes in stable/13 that are not in releng/13.2 > >>>> are the "make it work in a jail" stuff. Unfortunately, they are > >>>> a large # of changes (mostly trivial edits adding vnet macros), > >>>> but it also includes export check changes. > >>>> > >>>> I have attached a trivial patch that I think disables the export > >>>> checks for jails. If either of you can try it and see if it fixes > >>>> the problem, that would be great. > >>>> (Note that this is only for testing, although it probably does not > >>>> matter unless you are running nfsd(8) in vnet jails.) > >>> > >>> Yes, I can see snapshots with the patch. This system is just a test > >>> system that doesn't normally run ZFS or NFS, so no problem messing > >>> with permissions. It's a bhyve VM, so I just added a small disk and > >>> enabled ZFS for testing. > >> > >> btw, you might try to get mm@ or maybe mav@ to help out from the ZFS > >> side. It must be doing something differently inside a snapshot than > >> outside, maybe with file handles or something like that. > > Yes. I've added freebsd-current@ (although Garrett is not on it, he is > > cc'd) and these guys specifically... > > > > So, here's what appears to be the problem... > > Commit 88175af (in main and stable/13, but not 13.2) added checks for > > nfsd(8) running in jails by filling in mnt_exjail with a reference to t= he cred > > used when the file system is exported. > > When mnt_exjail is found NULL, the current nfsd code assumes that there > > is no access allowed for the mount. > > > > My vague understanding is that when a ZFS snapshot is accessed, it is > > "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > > mnt_exjail is NULL as a result. > > Since I do not know the ZFS code and don't even have an easy way to > > test this (thankfully Mike can test easily), I do not know what to do f= rom > > here? > > > > Is there a "struct mount" constructed for this pseudo mount > > (or it actually appears to be the lookup of ".." that fails, so it > > might be the parent of the snapshot subdir?)? > > > > One thought is that I can check to see if the mount pointer is in the > > mountlist (I don't think the snapshot's mount is in the mountlist) and > > avoid the jail test for this case. This would assume that snapshots ar= e > > always within the file system(s) exported via that jail (which includes > > the case of prison0, of course), so that they do not need a separate > > jail check. > > > > If this doesn't work, there will need to be some sort of messing about > > in ZFS to set mnt_exjail for these. > > > > I will try and get a test setup going here, which leads me to.. > > how do I create a ZFS snapshot? (I do have a simple ZFS pool running > > on a test machine, but I've never done a snapshot.) > > # zfs list > ... > zroot/usr/local 4.59G 27.5G 2.76G /usr/loca= l > zroot/usr/ports 1.03G 27.5G 952M /usr/port= s > ... > > # zfs snapshot zroot/usr/local@myfirstsnapshot > -- to view them > # zfs list -t snapshot zroot/usr/local > -- and to remove it: > # zfs destroy zroot/usr/local@myfirstsnapshot > -- more info > # man zfs-snapshot > > If you get used to this you are going to love it. :-) Not likely. My test systems are old laptops and don't need fancy storage. However, I do have one very simple setup for testing. To give you a clue, it's called /example because the description I found to do it used "example" and I didn't even realize it would end up as the name of the mount. However, thanks to Mike and others, I do now have a snapshot on it. Now, as noted in my other post, comes the hard part. I hope I can identify that the "mount is special and refers to a snapshot" from the *mp, so I can avoid the mp->mnt_exjail =3D=3D NULL check for this case. I'd like to avoid having to get ZFS set mnt_exjail for the snapshots. Thanks, rick > > Regards and happy hacking, > Ronald. > > > > > > Although this problem is not in 13.2, it will have shipped in 14.0. > > > > Any help with be appreciated, rick > > > >> > >> Mike > >>> > >>>> rick > >>>> > >>>> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > >>>>> > >>>>> CAUTION: This email originated from outside of the University of Gu= elph. Do not click links or open attachments unless you recognize the sende= r and know the content is safe. If in doubt, forward suspicious emails to I= Thelp@uoguelph.ca. > >>>>> > >>>>> > >>>>> Rick, have you been following this thread on freebsd-stable? I hav= e been able > >>>>> to reproduce this using a 13-stable server from Oct 7 and a 15-curr= ent system > >>>>> that is up to date using NFSv3. I did not reproduce with a 13.2 se= rver. The > >>>>> client was running 13.2. Any ideas? A full bisect seems fairly pa= inful, but > >>>>> maybe you have an idea of points to try. Fortunately, these are al= l test > >>>>> systems that I can reboot at will. > >>>>> > >>>>> Mike > >>>>> > >>>>> Forwarded message: > >>>>> > >>>>>> From: Garrett Wollman > >>>>>> To: Mike Karels > >>>>>> Cc: freebsd-stable@freebsd.org > >>>>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>>>> > >>>>>> < said: > >>>>>> > >>>>>>> I have not run into this, so I tried it just now. I had no probl= em. > >>>>>>> The server is 13.2, fully patched, the client is up-to-date -curr= ent, > >>>>>>> and the mount is v4. > >>>>>> > >>>>>> On my 13.2 client and 13-stable server, I see: > >>>>>> > >>>>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > >>>>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > >>>>>> 25034 ls RET open 4 > >>>>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>>>> 25034 ls RET fcntl 0 > >>>>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0x2= 37d32fa7028) > >>>>>> 25034 ls RET getdirentries -1 errno 5 Input/output error > >>>>>> 25034 ls CALL close(0x4) > >>>>>> 25034 ls RET close 0 > >>>>>> 25034 ls CALL exit(0) > >>>>>> > >>>>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > >>>>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > >>>>>> consistent across both FreeBSD and Linux clients. > >>>>>> > >>>>>> Looking at this from the RPC side: > >>>>>> > >>>>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>>>> [NFS4_OK for all ops] > >>>>>> (PUTFH, GETATTR) > >>>>>> [NFS4_OK, NFS4_OK] > >>>>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>>>> > >>>>>> and at this point the [EIO] is returned. > >>>>>> > >>>>>> It seems that clients always do a LOOKUPP before calling READDIR, = and > >>>>>> this is failing when the subject file handle is the snapshot. The > >>>>>> client is perfectly able to *traverse into* the snapshot: if I try= to > >>>>>> list a subdirectory I know exists in the snapshot, the client is a= ble to > >>>>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>>>> > >>>>>> -GAWollman > >>>>> > > >