Re: Speed improvements in ZFS

On 8/18/23, Alexander Leidinger <Alexander@leidinger.net> wrote:
> Am 2023-08-16 18:48, schrieb Alexander Leidinger:
>> Am 2023-08-15 23:29, schrieb Mateusz Guzik:
>>> On 8/15/23, Alexander Leidinger <Alexander@leidinger.net> wrote:
>>>> Am 2023-08-15 14:41, schrieb Mateusz Guzik:
>>>>
>>>>> With this in mind can you provide: sysctl kern.maxvnodes
>>>>> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes
>>>>> vfs.recycles_free vfs.recycles
>>>>
>>>> After a reboot:
>>>> kern.maxvnodes: 10485760
>>>> vfs.wantfreevnodes: 2621440
>>>> vfs.freevnodes: 24696
>>>> vfs.vnodes_created: 1658162
>>>> vfs.numvnodes: 173937
>>>> vfs.recycles_free: 0
>>>> vfs.recycles: 0
>>
>> New values after one rund of periodic:
>> kern.maxvnodes: 10485760
>> vfs.wantfreevnodes: 2621440
>> vfs.freevnodes: 356202
>> vfs.vnodes_created: 427696288
>> vfs.numvnodes: 532620
>> vfs.recycles_free: 20213257
>> vfs.recycles: 0
>
> And after the second round which only took 7h this night:
> kern.maxvnodes: 10485760
> vfs.wantfreevnodes: 2621440
> vfs.freevnodes: 3071754
> vfs.vnodes_created: 1275963316
> vfs.numvnodes: 3414906
> vfs.recycles_free: 58411371
> vfs.recycles: 0
>
>>>>> Meanwhile if there is tons of recycles, you can damage control by
>>>>> bumping kern.maxvnodes.
>>
>> What's the difference between recycles and recycles_free? Does the
>> above count as bumping the maxvnodes?
>
> ^^^^^
>
>>>> Looks like there are not much free directly after the reboot. I will
>>>> check the values tomorrow after the periodic run again and maybe
>>>> increase by 10 or 100 so see if it makes a difference.
>>>>
>>>>> If this is not the problem you can use dtrace to figure it out.
>>>>
>>>> dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or
>>>> something else?
>>>>
>>>
>>> I mean checking where find is spending time instead of speculating.
>>>
>>> There is no productized way to do it so to speak, but the following
>>> crapper should be good enough:
>> [script]
>>
>> I will let it run this night.
>
> I have a 51MB text file, compressed to about 1MB. Are you interested to
> get it?
>

Your problem is not the vnode limit, but nullfs.

https://people.freebsd.org/~mjg/netchild-periodic-find.svg

First, some of the contention is notorious VI_LOCK in order to do anything.

But more importantly the mind-boggling off-cpu time comes from
exclusive locking which should not be there to begin with -- as in
that xlock in stat should be a slock.

Maybe I'm going to look into it later.

-- 
Mateusz Guzik <mjguzik gmail.com>