Re: sysctl is too slow
- In reply to: Alan Somers : "Re: sysctl is too slow"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 08 Sep 2022 23:30:31 UTC
I finally got around to this. Here's the review: https://reviews.freebsd.org/D36500 On Mon, Aug 23, 2021 at 6:54 AM Alan Somers <asomers@freebsd.org> wrote: > > Ideally, but it's not very high priority, since it's merely a performance issue in a monitoring tool. > > On Mon, Aug 23, 2021 at 6:05 AM Mateusz Guzik <mjguzik@gmail.com> wrote: >> >> So is this something you plan on fixing? >> >> On 8/17/21, Alan Somers <asomers@freebsd.org> wrote: >> > Actually, I did get a flamegraph, and only 0.77% of samples were in ZFS. >> > >> > On Mon, Aug 16, 2021 at 7:19 PM Mateusz Guzik <mjguzik@gmail.com> wrote: >> > >> >> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote: >> >> > Yes, I see what you're talking about now. There are a bunch of linked >> >> > lists in sysctl_find_oid etc. Good point. >> >> > -Alan >> >> > >> >> >> >> You still want to get a flamegraph, chances are most of the problem is in >> >> zfs. >> >> >> >> > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik <mjguzik@gmail.com> >> >> > wrote: >> >> > >> >> >> Last time I checked lookup of a sysctl was very bad with linear scans >> >> all >> >> >> over. >> >> >> >> >> >> Short of complete revamp of the entire thing I would start with >> >> >> replacing the scans with a RB tree at each level. As is if you indeed >> >> >> have 5000 datasets, you are doing increasingly longer walks. >> >> >> >> >> >> On 8/16/21, Alan Somers <asomers@freebsd.org> wrote: >> >> >> > ztop feels very sluggish on a server with 5000 ZFS datasets. Dtrace >> >> >> shows >> >> >> > that almost all of its time is spent in sys_sysctl. ktrace shows >> >> >> > that >> >> >> both >> >> >> > ztop and sysctl(8) call sys_sysctl a total of five times for each >> >> >> > sysctl >> >> >> > they care about: >> >> >> > >> >> >> > 1) To get the next oid >> >> >> > 2) To get the sysctl's name >> >> >> > 3) To get the oidfmt >> >> >> > 4) To get the size of the value >> >> >> > 5) To get the value itself. >> >> >> > >> >> >> > Each of these steps takes about equal time, and together all five >> >> >> > take >> >> >> > about 100us. If the time per call is mostly syscall overhead, then >> >> the >> >> >> > process could be sped up by 80% by combining all of these things >> >> >> > into >> >> a >> >> >> > single syscall: return the next oid, its name, its format, the size >> >> >> > of >> >> >> its >> >> >> > value, and optimistically the value itself, assuming the user passed >> >> >> > a >> >> >> > sufficiently large buffer. >> >> >> > >> >> >> > Am I missing something? Is there any other reason why sysctl is so >> >> >> > slow? >> >> >> > Or should I forget about it, and try to export ZFS's dataset stats >> >> >> through >> >> >> > devstat instead? >> >> >> > -Alan >> >> >> > >> >> >> >> >> >> >> >> >> -- >> >> >> Mateusz Guzik <mjguzik gmail.com> >> >> >> >> >> > >> >> >> >> >> >> -- >> >> Mateusz Guzik <mjguzik gmail.com> >> >> >> > >> >> >> -- >> Mateusz Guzik <mjguzik gmail.com>