From nobody Sat Nov 18 21:58:20 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXnhm21TNz50hfR for ; Sat, 18 Nov 2023 21:58:32 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oa1-x35.google.com (mail-oa1-x35.google.com [IPv6:2001:4860:4864:20::35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXnhl32jzz4cNb; Sat, 18 Nov 2023 21:58:31 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=kguMvSgQ; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2001:4860:4864:20::35 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x35.google.com with SMTP id 586e51a60fabf-1f00b95dc43so1803923fac.3; Sat, 18 Nov 2023 13:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700344710; x=1700949510; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HjMlC5JAhisItwQUfd8YDopK5d2PPwiiK4qvs+2jZ3I=; b=kguMvSgQh2UavZFu5HlXGACbkrQSXW2x5Fv2gmYj9HJVimkd+iZzNltRTqV5AGwXXL 3zVKQ6Ogb+dAhH2jCjEfSqu9p+psExrYU8msacoLDKCq+FgYDRGCAIn4FihBXuhO2z8H tVyU4uDza0g9oq7xnZUPPtxWtSsUA54vPcnI9bPKwA+MvLchnx3C9T3QxYi+mbQPnMfz fdOr4Y+kXPNmwQGkUFP2fHp/uwS1yWNlC4vNKLRdeipIRV8KFuPMap5cOJ/gxwBBr5J/ SNuWsV++fMmEjHXJYw5Hd+uWVdAiuM7xVK2RkBmDFCBG68PTFSc3WZ9RxVMBkSBJzpSt 2zog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700344710; x=1700949510; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HjMlC5JAhisItwQUfd8YDopK5d2PPwiiK4qvs+2jZ3I=; b=LGht0/QDZA+/dQldBPK4TZNNpne/dEA5helLsc2Z8irGIVBMXokO08NUnLhj3kBaZ2 N+4YAu2we9AcmZV3BtqU00RzjqN5jgs7Hj6ojGpiOZQ+/zr8ndSGes1wE4f8pfsVsDhz ixrsEouRBc2ZIb5x5BLmb7oaW0xt1I4GNCcmhKYczYVXChFD9p2nqroxSZAw9rpIiBXy 2U9PLt5LDVmtB1Ff26AItysNh4aApRLdo5gkEm1dXtB9cBFrNkPwDcFZcqBvTjJT08tb E8Z91CeW3EKrv2D/r5XBOxRPsUe9e5oZBq8uh/xeTcK/K3Ybt6t5/we9isZs02uy/ynO ivog== X-Gm-Message-State: AOJu0YxFUytH3A6HCafwlAMWEKO80Kkdxadta+8BsPGJolypyzAWRvJC /DAAQMJRYVwfPlTG5nXlPqf1ODg3LWMMYSB+JQ5LMPo= X-Google-Smtp-Source: AGHT+IEBKKWrrJUSvpQVPzVkhZMHrUC6uOFPSPbiyeFtsAKyQCoUPgqBm9FRvUDv1KIiERUb/bc4Qg3tzUnNV6evJcE= X-Received: by 2002:a05:6870:a40b:b0:1e9:cde8:6db0 with SMTP id m11-20020a056870a40b00b001e9cde86db0mr4047109oal.50.1700344710075; Sat, 18 Nov 2023 13:58:30 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sat, 18 Nov 2023 13:58:20 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels , FreeBSD CURRENT Cc: Rick Macklem , Garrett Wollman , Alexander Motin , Martin Matuska Content-Type: multipart/mixed; boundary="00000000000072469f060a745a22" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; TAGGED_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::35:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; HAS_ATTACHMENT(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org] X-Rspamd-Queue-Id: 4SXnhl32jzz4cNb X-Spamd-Bar: --- --00000000000072469f060a745a22 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem wrote: > > On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wro= te: > > > > On 17 Nov 2023, at 22:14, Mike Karels wrote: > > > > > On 17 Nov 2023, at 21:24, Rick Macklem wrote: > > > > > >> Most of the changes in stable/13 that are not in releng/13.2 > > >> are the "make it work in a jail" stuff. Unfortunately, they are > > >> a large # of changes (mostly trivial edits adding vnet macros), > > >> but it also includes export check changes. > > >> > > >> I have attached a trivial patch that I think disables the export > > >> checks for jails. If either of you can try it and see if it fixes > > >> the problem, that would be great. > > >> (Note that this is only for testing, although it probably does not > > >> matter unless you are running nfsd(8) in vnet jails.) > > > > > > Yes, I can see snapshots with the patch. This system is just a test > > > system that doesn't normally run ZFS or NFS, so no problem messing > > > with permissions. It's a bhyve VM, so I just added a small disk and > > > enabled ZFS for testing. > > > > btw, you might try to get mm@ or maybe mav@ to help out from the ZFS > > side. It must be doing something differently inside a snapshot than > > outside, maybe with file handles or something like that. > Yes. I've added freebsd-current@ (although Garrett is not on it, he is > cc'd) and these guys specifically... > > So, here's what appears to be the problem... > Commit 88175af (in main and stable/13, but not 13.2) added checks for > nfsd(8) running in jails by filling in mnt_exjail with a reference to the= cred > used when the file system is exported. > When mnt_exjail is found NULL, the current nfsd code assumes that there > is no access allowed for the mount. > > My vague understanding is that when a ZFS snapshot is accessed, it is > "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that > mnt_exjail is NULL as a result. > Since I do not know the ZFS code and don't even have an easy way to > test this (thankfully Mike can test easily), I do not know what to do fro= m > here? > > Is there a "struct mount" constructed for this pseudo mount > (or it actually appears to be the lookup of ".." that fails, so it > might be the parent of the snapshot subdir?)? > > One thought is that I can check to see if the mount pointer is in the > mountlist (I don't think the snapshot's mount is in the mountlist) and > avoid the jail test for this case. This would assume that snapshots are > always within the file system(s) exported via that jail (which includes > the case of prison0, of course), so that they do not need a separate > jail check. > > If this doesn't work, there will need to be some sort of messing about > in ZFS to set mnt_exjail for these. Ok, so now onto the hard part... Thanks to Mike and others, I did create a snapshot under .zfs and I can see the problem. It is that mnt_exjail =3D=3D NULL. Now, is there a way that this "struct mount" can be recognized as "special" for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test? (I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is not the case.) Do I need to search mountlist for it? rick ps: The hack patch attached should fix the problem, but can only be safely used if mountd/nfsd are not run in any jails. > > I will try and get a test setup going here, which leads me to.. > how do I create a ZFS snapshot? (I do have a simple ZFS pool running > on a test machine, but I've never done a snapshot.) > > Although this problem is not in 13.2, it will have shipped in 14.0. > > Any help with be appreciated, rick > > > > > Mike > > > > > >> rick > > >> > > >> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels wrote: > > >>> > > >>> CAUTION: This email originated from outside of the University of Gu= elph. Do not click links or open attachments unless you recognize the sende= r and know the content is safe. If in doubt, forward suspicious emails to I= Thelp@uoguelph.ca. > > >>> > > >>> > > >>> Rick, have you been following this thread on freebsd-stable? I hav= e been able > > >>> to reproduce this using a 13-stable server from Oct 7 and a 15-curr= ent system > > >>> that is up to date using NFSv3. I did not reproduce with a 13.2 se= rver. The > > >>> client was running 13.2. Any ideas? A full bisect seems fairly pa= inful, but > > >>> maybe you have an idea of points to try. Fortunately, these are al= l test > > >>> systems that I can reboot at will. > > >>> > > >>> Mike > > >>> > > >>> Forwarded message: > > >>> > > >>>> From: Garrett Wollman > > >>>> To: Mike Karels > > >>>> Cc: freebsd-stable@freebsd.org > > >>>> Subject: Re: NFS exports of ZFS snapshots broken > > >>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > > >>>> > > >>>> < said: > > >>>> > > >>>>> I have not run into this, so I tried it just now. I had no probl= em. > > >>>>> The server is 13.2, fully patched, the client is up-to-date -curr= ent, > > >>>>> and the mount is v4. > > >>>> > > >>>> On my 13.2 client and 13-stable server, I see: > > >>>> > > >>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > > >>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > > >>>> 25034 ls RET open 4 > > >>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > > >>>> 25034 ls RET fcntl 0 > > >>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0x23= 7d32fa7028) > > >>>> 25034 ls RET getdirentries -1 errno 5 Input/output error > > >>>> 25034 ls CALL close(0x4) > > >>>> 25034 ls RET close 0 > > >>>> 25034 ls CALL exit(0) > > >>>> > > >>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > > >>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > > >>>> consistent across both FreeBSD and Linux clients. > > >>>> > > >>>> Looking at this from the RPC side: > > >>>> > > >>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > > >>>> [NFS4_OK for all ops] > > >>>> (PUTFH, GETATTR) > > >>>> [NFS4_OK, NFS4_OK] > > >>>> (PUTFH, ACCESS(0x3f), GETATTR) > > >>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > > >>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > > >>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > > >>>> > > >>>> and at this point the [EIO] is returned. > > >>>> > > >>>> It seems that clients always do a LOOKUPP before calling READDIR, = and > > >>>> this is failing when the subject file handle is the snapshot. The > > >>>> client is perfectly able to *traverse into* the snapshot: if I try= to > > >>>> list a subdirectory I know exists in the snapshot, the client is a= ble to > > >>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > > >>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > > >>>> > > >>>> -GAWollman > > >>> --00000000000072469f060a745a22 Content-Type: application/octet-stream; name="zfsjail.patch" Content-Disposition: attachment; filename="zfsjail.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lp4lbu7d0 LS0tIHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2Rwb3J0LmMuamFpbAkyMDIzLTExLTE3IDE5OjEw OjEyLjg3MTYxOTAwMCAtMDgwMAorKysgc3lzL2ZzL25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYwky MDIzLTExLTE3IDE5OjExOjM4Ljk0MjY3NDAwMCAtMDgwMApAQCAtMzI4MiwxMSArMzI4MiwxMyBA QCBuZnN2bm9fY2hlY2tleHAoc3RydWN0IG1vdW50ICptcCwgc3RydWN0IHNvY2thZGRyICpuYW0K IAogCWVycm9yID0gMDsKIAkqY3JlZHAgPSBOVUxMOworI2lmZGVmIG5vdG5vdwogCU1OVF9JTE9D SyhtcCk7CiAJaWYgKG1wLT5tbnRfZXhqYWlsID09IE5VTEwgfHwKIAkgICAgbXAtPm1udF9leGph aWwtPmNyX3ByaXNvbiAhPSBjdXJ0aHJlYWQtPnRkX3VjcmVkLT5jcl9wcmlzb24pCiAJCWVycm9y ID0gRUFDQ0VTOwogCU1OVF9JVU5MT0NLKG1wKTsKKyNlbmRpZgogCWlmIChlcnJvciA9PSAwKQog CQllcnJvciA9IFZGU19DSEVDS0VYUChtcCwgbmFtLCAmZXhwLT5uZXNfZXhmbGFnLCBjcmVkcCwK IAkJICAgICZleHAtPm5lc19udW1zZWNmbGF2b3IsIGV4cC0+bmVzX3NlY2ZsYXZvcnMpOwpAQCAt MzMyMywxMSArMzMyNSwxMyBAQCBuZnN2bm9fZmh0b3ZwKHN0cnVjdCBtb3VudCAqbXAsIGZoYW5k bGVfdCAqZmhwLCBzdHJ1Y3QKIAkJLyogTWFrZSBzdXJlIHRoZSBzZXJ2ZXIgcmVwbGllcyBFU1RB TEUgdG8gdGhlIGNsaWVudC4gKi8KIAkJZXJyb3IgPSBFU1RBTEU7CiAJaWYgKG5hbSAmJiAhZXJy b3IpIHsKKyNpZmRlZiBub3Rub3cKIAkJTU5UX0lMT0NLKG1wKTsKIAkJaWYgKG1wLT5tbnRfZXhq YWlsID09IE5VTEwgfHwKIAkJICAgIG1wLT5tbnRfZXhqYWlsLT5jcl9wcmlzb24gIT0gY3VydGhy ZWFkLT50ZF91Y3JlZC0+Y3JfcHJpc29uKQogCQkJZXJyb3IgPSBFQUNDRVM7CiAJCU1OVF9JVU5M T0NLKG1wKTsKKyNlbmRpZgogCQlpZiAoZXJyb3IgPT0gMCkKIAkJCWVycm9yID0gVkZTX0NIRUNL RVhQKG1wLCBuYW0sICZleHAtPm5lc19leGZsYWcsIGNyZWRwLAogCQkJICAgICZleHAtPm5lc19u dW1zZWNmbGF2b3IsIGV4cC0+bmVzX3NlY2ZsYXZvcnMpOwo= --00000000000072469f060a745a22--