Re: FreeBSD 12.3/13.1 NFS client hang

Kurt Jaeger <pi@freebsd.org> wrote:
>
> Hi!
>
> > > Kurt Jaeger <pi@freebsd.org> wrote:
> > > > > > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1

> > > > I have it with an 13.0p7 client against an 13.1 server with
> > > > a hanging soft-mount (I tried unmount to change it to a hard mount).
[...]
> > I take this back. I just did a fairly trivial test of this and it worked.
>
> > Looking at the "ps" output, I don't think your case is a "NFS protocol hang".
>
> Today I upgraded the client to 13.1, and changed the nfs mount
> from soft to hard,intr.
>
> nfsstat -M now show this:
>
> # nfsstat -m
> <myhost>:/serv on /office/serv
>nfsv3,tcp,resvport,nconnect=1,hard,intr,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2
>
Since you are using NFSv3 mounts, the comments w.r.t. hard vs soft are
not relevant.

> > We use RCS for single files, quite a lot. Sometimes the co -l or the
> > ci -u would not return and hang as well.
>
> We still have rpc.lockd and rpc.statd running on the server,
> are they still used or not ? What happens.
Since you are using NFSv3, if you want the locks to be visible across
multiple clients then, yes, you need to run these.
Alternately, you can use the "nolockd" mount option, which avoids
using rpc.lockd and creates file locks visible within the client only.
(NFSv4 does not use rpc.lockd, rpc.statd.)

> We also use vi to edit files, and in the past they reliably showed
> us when someone on nfsclient A had a file in vi and someone on
> nfsclient B tried to edit the same file.
> 
> Let's see if the problems re-occurs with hard mounts.
It probably will, since you are using NFSv3 and, as I said, the hang you
posted "ps" output for was not even an NFS problem.
--> It appears to be a VM problem.

> I'll investigate how to move to nfsv4 with a test box as well, if
> this helps ?
Again, I do not think this is an NFS problem, so switching to NFSv4
will not resolve this.

> > I suspect that some process/thread is hung for something non-NFS
> > while holding a lock on a NFS vnode. "umount -N" won't know how
> > to unhang this process/thread.
> 
> > Just a hunch, but I'd suspect one of the threads sleeping on
> > "vmopar", although I'm not a vm guy.
> > What I don't know how to do is figure out what thread(s) are
> > holding vnode locks?
>
> Given that simple NFS mounts and editing with vi on nfs mounts should
> be very basic use-cases, it's time to find ways to debug this 8-(
Look for a VM problem...

> This also implies that switching from soft->hard won't fix the problem.
>
> Thanks for the encouragement 8-}
>
> > In summary, I don't think your hang is anything like Andreas's, rick
>
> I'll report as soon as I have a another hang.
I suggest you report this to a mailing list where the VM guys hang
out, with a subject line like "processes hung sleeping on vmopar".
Don't even mention NFS in the subject line. Reporting it here with
NFS in the subject line won't get it seem by the VM people.
(I could be wrong w.r.t. it being a VM problem, but I am pretty
 sure it is not an NFS problem. It just happens to result in NFS
 vnode being locked, which results in the hangs you observe.)

rick

--
pi@FreeBSD.org         +49 171 3101372                  Now what ?