Re: RFC: nfsd in a vnet jail

On Fri, Nov 25, 2022 at 9:06 PM Alan Somers <asomers@freebsd.org> wrote:

>
>
> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>
>> Hi,
>>
>> bz@ has encouraged me to fiddle with the nfsd
>> so that it works in a vnet jail.
>> I have now basically done so, specifically for
>> NFSv4, since NFSv3 presents various issues.
>>
>> What I have not yet done is put global variables
>> in the vnet. This needs to be done so that the nfsd
>> can be run in multiple jail instances and/or in and
>> outside of a jail.
>> The problem is that there are 100s of global variables.
>>
>> I can see two approaches:
>> 1 - Move them all into the vnet jail. This would imply
>>     that all the sysctls need to somehow be changed,
>>     which would seem to be a POLA violation.
>>     It also implies a lot of stuff in the vnet.
>> 2 - Just move the global variables that will always
>>     differ from one nfsd to another (this would make
>>     the sysctls global and apply to all nfsds).
>>     This will keep the number of globals in the vnet
>>     smaller.
>>
>> I am currently leaning towards #2, put what do others
>> think?
>>
>> rick
>> ps: Personally, I don't know what use there is of
>>     running the nfsd inside a vnet jail, but bz@ has
>>     some use case.
>>
>
> This is super-awesome! Thank you so much! I've got a use case too.  I
> think it would be fine to leave most of the settings global,  like
> max_threads. But we should probably decide on a case by case basis .
>
The minthreads, maxthreads happen to be handled via nfsd command line
options, so
the sysctls are not needed and they can be set per-prison.
Most of the sysctls are for weird cases or tuning of the DRC. Since the DRC
is
only used for NFSv4.0 mounts and not NFSv4.1 or NFSv4.2 ones, tuning the DRC
should not usually be necessary.

I have left them global for now.

If anyone identifies one that needs to be set per-prison, I can move it into
the vnet.
If you want to see them all:
# sysctl -a | fgrep vfs.nfsd

I have put a first patch up on phabricator as D37519. Although I listed
three
people as reviewers, anyone is welcome to test/comment/review.
If you can't easily get the patch from phabricator, just email me and I'll
send it to you. I think it will apply cleanly to main and, maybe, stable/13.
You only need to build a kernel from patched sources to test it. There is a
change to rc.d/nfsd, which you only need in the prison's etc/rc.d/nfsd.

A very basic setup document (also definitely a work in progress) can be
found at...
https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt

Let me know if you test it or have other suggestions, rick
ps: Thanks everyone for your comments. If I have specific questions related
    to them, I'll post. Otherwise I am digesting them.