Major issues with nfsv4

[Rick Macklem]

J David wrote:
>On Thu, Jan 14, 2021 at 5:30 PM Rick Macklem <rmacklem at uoguelph.ca> wrote:
>> One thing to try (other than a FreeBSD13/head system, if possible)
>> is the "oneopenown" mount option.
>
>The odds of being able to run an unreleased version of FreeBSD on
>production servers are slim to none.
Well, the current release schedule has FreeBSD13 being released at the
end of March. If you set up a test system now/soon, you might be able
to determine if an upgrade to FreeBSD13 is justified when it is released?

>While trying to develop a reproduction, I think I have narrowed down
>what the problem is.  There are no jails or nullfs involved here, just
>NFSv4.1.
>
>Window 1: (to track OpenOwner/Opens)
>
>while true; do date; nfsstat -E -c | fgrep -A1 OpenOwner; sleep 1; done
>
>Window 2:
>
>mount -o ro,nfsv4,minorversion=1,nosuid fileserver:/path/to/freesbd/root /mnt
>chroot /mnt
>
>(OpenOwner is now 1 and Opens is now 9.)
>
>Window 3:
>
>chroot /mnt
>(OpenOwner is now 2 and Opens is now 18.)
>ls
>(OpenOwner is now 3 and Opens is now 21.)
>ls
>(OpenOwner is now 4 and Opens is now 24.)
>ls
>(OpenOwner is now 5 and Opens is now 27.)
>ls
>(OpenOwner is now 6 and Opens is now 30.)
>ls
>(OpenOwner is now 7 and Opens is now 33.)
>bash
>while true; do ls | true; done
>(Allow about a minute to pass, hit CTRL-C.  OpenOwner is now 4647 and
>Opens is now 13957)
>exit
>exit
>(OpenOwner is now 4647 and Opens is now 13952.)
Hmm. Not sure what files would get opened each time. NFSv4 Opens
only apply to regular files, so "ls" shouldn't result in Opens.

I may try this and see what the 3 files being Open'd are.
(Obviously something related to the chroot. But what?)

>Back in Window 2:
>
>exit
>(wait about 30 seconds, OpenOwner is now 0 and Opens is now 0.)
Yes, it could take a while to close all those opens.

>So it looks like the NFSv4 code can't let go of *any* Opens on a
>file/directory until *all* references to that file/directory are
>closed.
True for regular files, but not directories.

>If chroot is too much, "vi /mnt/etc/motd" in Window 2 and "cat
>/mnt/etc/motd" in Window 3 have the same effect, leaking one Open per
>cat instead of 3.  You probably don't even need a FreeBSD install on
>the NFS mount; just hold a single file open in one window and
>open/close it repeatedly in another.
An NFSv4 Open is unique for open_owner/file, so the same file opened
by different processes (an openowner represents a process unless you
use "oneopenown") results in separate NFSv4 Opens.
However, since the NFSv4 client cannot know which Open a VOP_CLOSE()
is associated with, due to file descriptor inheritance, none of the NFSv4
Opens can be closed until all FreeBSD open file descriptors for the file
are closed.
--> Just the way it is. It is not an unintended leak. They go away once
      all file descriptors get closed, so long as the VOP_INACTIVE() gets
      called for the NFSv4 vnode.
      --> It is the handling of deferred VOP_INACTIVE() calls that has
             changed for FreeBSD13.
However, none of the above seems unexpected, except maybe for why
"ls" in the chroot opens 3 regular files each time. I don't know what
chroot actually does for something like "ls"? I'll look.

>Then I re-tested this with "-o
>ro,nfsv4,minorversion=1,nosuid,oneopenown."  At least for this simple
>case, the problem did not occur with oneopenown set.
Yes. For the oneopenown case, there will only be one NFSv4 Open for
each file opened.

>Are there downsides to the oneopenown flag other than breaking delegations?
Here's an example:
- One process running as J David opens a file for reading, which works since
   J David has read permissions on the file.
- Another process running as Warner opens the same file for writing, which
  works, since Warner has write access for the file.

Now, network partition the client from the server until the lease expires...
The client now gets a NFSERR_EXPIRED error, which forces it to retry the
open(s).

Without "oneopenown", the above FreeBSD opens resulted in 2 NFSv4
Opens, both of which probably reopen successfully (unless a chmod/chown/
setfacl on the file makes the reopen fail).

With "oneopenown", the above FreeBSD opens results in one NFSv4 Open
for reading/writing. A retry of this one Open might succeed, depending on
what the file permissions are. (I think the code uses the credentials for
Warner in this case, assuming that the credentials that opened it for
writing is more likely to succeed, but there are no guarantees.

--> For normal operation, it should be fine. A network partition that
      results in NFSERR_EXPIRED is a worst case scenario, where all
      byte range locks will be lost and Opens may be lost.

For delegations, the story is similar, but happens routinely when
delegations are recalled by the server.
--> For delegation recall, the reopen is done using a special variant of Open called
       claim_delegate_current. A server should allow claim_delegate_current
       irrespective of what the file permissions are, but the original RFC3530
       did not make this clear.
       --> Is allowed by a FreeBSD NFSv4 server.
As such, the warning in the man page is mainly there for NFSv4 servers
where the reopen done at delegation recall time can fail, due to permission
checking.

There is also the fact that the case of "oneopenown+delegations" is not well
tested and the current FreeBSD client code will use separate open_owners
(violating the "one open owner" principal) when delegations are recalled.

Are delegations useful?
- Short answer, often not.

Delegations allow the client to do 2 things:
1 - More extensive file data caching, since the delegation guarantees that
      other clients will not be modifying the file.
      This is not exploited by current clients, as far as I know. I had something
      called Packrat, but it never made it into FreeBSD. More on Packrat below,
      for anyone interested.
2 - Do NFSv4 Opens locally in the client, avoiding the Open/Close RPCs.
     Since delegations are per file, this only helps if the same files get opened
     over and over and over again. Doesn't happen for many loads, from what
     I've seen.
     --> With delegations turned on, you can compare the counts for Opens
           vs LocalOpens in the "nfsstat -E -c" stats, to see how many Open/Close
           RPCs get saved.

Packrats: Some now bitrotted code in the subversion projects area that did
              whole file caching of small files in non-volatile storage on the client
              when a delegation was acquired for the file.
              This was intended for devices like laptops, with slow/flakey network
              connectivity.
              --> Recent FreeBSD changes might inspire me to resurrect this.
                    - FreeBSD has recently become more laptop friendly.
                    - nfs-over-tls provides an easy way to make NFSv4 mounts
                      from anywhere (as a laptop might) relatively secure.

rick

Thanks!