Re: Speed improvements in ZFS

On 8/18/23, Alexander Leidinger <Alexander@leidinger.net> wrote:
> Am 2023-08-16 18:48, schrieb Alexander Leidinger:
>> Am 2023-08-15 23:29, schrieb Mateusz Guzik:
>>> On 8/15/23, Alexander Leidinger <Alexander@leidinger.net> wrote:
>>>> Am 2023-08-15 14:41, schrieb Mateusz Guzik:
>>>>
>>>>> With this in mind can you provide: sysctl kern.maxvnodes
>>>>> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes
>>>>> vfs.recycles_free vfs.recycles
>>>>
>>>> After a reboot:
>>>> kern.maxvnodes: 10485760
>>>> vfs.wantfreevnodes: 2621440
>>>> vfs.freevnodes: 24696
>>>> vfs.vnodes_created: 1658162
>>>> vfs.numvnodes: 173937
>>>> vfs.recycles_free: 0
>>>> vfs.recycles: 0
>>
>> New values after one rund of periodic:
>> kern.maxvnodes: 10485760
>> vfs.wantfreevnodes: 2621440
>> vfs.freevnodes: 356202
>> vfs.vnodes_created: 427696288
>> vfs.numvnodes: 532620
>> vfs.recycles_free: 20213257
>> vfs.recycles: 0
>
> And after the second round which only took 7h this night:
> kern.maxvnodes: 10485760
> vfs.wantfreevnodes: 2621440
> vfs.freevnodes: 3071754
> vfs.vnodes_created: 1275963316
> vfs.numvnodes: 3414906
> vfs.recycles_free: 58411371
> vfs.recycles: 0
>

so your setup has a vastly higher number of vnodes to inspect than the
number of vnodes it allows to exist at the same time, which further
suggests it easily may be about that msleep.

>>>>> Meanwhile if there is tons of recycles, you can damage control by
>>>>> bumping kern.maxvnodes.
>>
>> What's the difference between recycles and recycles_free? Does the
>> above count as bumping the maxvnodes?
>
> ^^^^^
>

"free" vnodes are just hanging around and can be directly whacked, the
others are used but *maybe* freeable (say a directory with a bunch of
vnodes already established).

>>>> Looks like there are not much free directly after the reboot. I will
>>>> check the values tomorrow after the periodic run again and maybe
>>>> increase by 10 or 100 so see if it makes a difference.
>>>>
>>>>> If this is not the problem you can use dtrace to figure it out.
>>>>
>>>> dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or
>>>> something else?
>>>>
>>>
>>> I mean checking where find is spending time instead of speculating.
>>>
>>> There is no productized way to do it so to speak, but the following
>>> crapper should be good enough:
>> [script]
>>
>> I will let it run this night.
>
> I have a 51MB text file, compressed to about 1MB. Are you interested to
> get it?
>

Yea, put it on freefall for example.

or feed it directly to flamegraph: cat file | ./stackcollapse.pl |
./flamegraph.pl > out.svg

see this repo https://github.com/brendangregg/FlameGraph.git


-- 
Mateusz Guzik <mjguzik gmail.com>