Re: sysctl is too slow

Actually, I did get a flamegraph, and only 0.77% of samples were in ZFS.

On Mon, Aug 16, 2021 at 7:19 PM Mateusz Guzik <mjguzik@gmail.com> wrote:

> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote:
> > Yes, I see what you're talking about now.  There are a bunch of linked
> > lists in sysctl_find_oid etc.  Good point.
> > -Alan
> >
>
> You still want to get a flamegraph, chances are most of the problem is in
> zfs.
>
> > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik <mjguzik@gmail.com> wrote:
> >
> >> Last time I checked lookup of a sysctl was very bad with linear scans
> all
> >> over.
> >>
> >> Short of complete revamp of the entire thing I would start with
> >> replacing the scans with a RB tree at each level. As is if you indeed
> >> have 5000 datasets, you are doing increasingly longer walks.
> >>
> >> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote:
> >> > ztop feels very sluggish on a server with 5000 ZFS datasets.  Dtrace
> >> shows
> >> > that almost all of its time is spent in sys_sysctl.  ktrace shows that
> >> both
> >> > ztop and sysctl(8) call sys_sysctl a total of five times for each
> >> > sysctl
> >> > they care about:
> >> >
> >> > 1) To get the next oid
> >> > 2) To get the sysctl's name
> >> > 3) To get the oidfmt
> >> > 4) To get the size of the value
> >> > 5) To get the value itself.
> >> >
> >> > Each of these steps takes about equal time, and together all five take
> >> > about 100us.  If the time per call is mostly syscall overhead, then
> the
> >> > process could be sped up by 80% by combining all of these things into
> a
> >> > single syscall: return the next oid, its name, its format, the size of
> >> its
> >> > value, and optimistically the value itself, assuming the user passed a
> >> > sufficiently large buffer.
> >> >
> >> > Am I missing something?  Is there any other reason why sysctl is so
> >> > slow?
> >> > Or should I forget about it, and try to export ZFS's dataset stats
> >> through
> >> > devstat instead?
> >> > -Alan
> >> >
> >>
> >>
> >> --
> >> Mateusz Guzik <mjguzik gmail.com>
> >>
> >
>
>
> --
> Mateusz Guzik <mjguzik gmail.com>
>