Re: 13-stable NFS server hang

On Sat, Mar 02, 2024 at 05:40:08AM -0800, Rick Macklem wrote:
> On Fri, Mar 1, 2024 at 10:51 PM Konstantin Belousov <kib@freebsd.org> wrote:
> >
> > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
> >
> >
> > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote:
> > > On Fri, Mar 1, 2024 at 12:00 AM Ronald Klop <ronald-lists@klop.ws> wrote:
> > > >
> > > > Interesting read.
> > > >
> > > >  Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations?
> > > Well, the NFS server does not really have any concept of a mount.
> > > What I am referring to is the ClientID maintained for NFSv4 mounts,
> > > which all the open/lock/session/layout state hangs off of.
> > >
> > > For most cases, this state information can safely be accessed/modified
> > > via a mutex, but there are three exceptions:
> > > - creating a new ClientID (which is done by the ExchangeID operation)
> > >   and typically happens when a NFS client does a mount.
> > > - delegation Recall (which only happens when delegations are enabled)
> > >   One of the reasons delegations are not enabled by default on the
> > > FreeBSD server.
> > > - the DestroyClientID which is typically done by a NFS client during dismount.
> > > For these cases, it is just too difficult to do them without sleeping.
> > > As such, there is a sleep lock which the nfsd threads normally acquire shared
> > > when doing NFSv4 operations, but for the above cases the lock is aquired
> > > exclusive.
> > > - I had to give the exclusive lock priority over shared lock
> > > acquisition (it is a
> > >   custom locking mechanism with assorted weirdnesses) because without
> > >   that someone reported that new mounts took up to 1/2hr to occur.
> > >   (The exclusive locker waited for 30min before all the other nfsd threads
> > >    were not busy.)
> > >   Because of this priority, once a nfsd thread requests the exclusive lock,
> > >   all other nfsd threads executing NFSv4 RPCs block after releasing their
> > >   shared lock, until the exclusive locker releases the exclusive lock.
> > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the
> > property of pref. exclusive waiters in presence of the shared waiters.
> > I think this is what you described above.
> It also has some weird properties, like if there are multiple requestors
> for the exclusive lock, once one thread gets it (the threads are nfsd worker
> threads and indistinct), the others that requested an exclusive thread are
> unblocked without the lock being issued to them.
This sounds to me as LK_SLEEPFAIL feature of lockmgr.
Do not underestimate the amount of weird features in it.

> They then check if the exclusive lock is still needed (usually not, since
> the other thread has dealt with the case where it was needed) and
> then they can acquire a shared lock.
> Without this, there were cases where several threads would acquire
> the exclusive lock and then discover that the lock was not needed and
> just release it again.
> 
> It also uses an assortment of weird flags/call args.
> 
> rick
>