From nobody Mon Aug 23 12:04:49 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6B9541777A4B for ; Mon, 23 Aug 2021 12:04:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GtWBy2xBJz58xV; Mon, 23 Aug 2021 12:04:58 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lj1-x22b.google.com with SMTP id q21so31037091ljj.6; Mon, 23 Aug 2021 05:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oE7F/pmLVdPW6Kss7G1myKGx46vWPSP6E/UNtsRt16o=; b=Gn3TkNwrXo17jz1LDoSsIIqg5UVd4qkgRYsiGbo3fCn0uFQwehcGAvNlOL79leNuUG a+H6jCkR3HedpiPSVZzp9Y3AyXLolv+pP8leQoj/fEuWDk4UWKbt8EjkTnLho+uhcyDV ll4uA2jiQX4/HhyMPURU/FUyzeF4I39K9Z+MAVlp8q6uRnI7HRvbM+CWhbD77Iuh6I3G OQsU1+72dRDtmgJvFMoXlmmrwarMdMt5Zd7qZCTDMHbBBxL+IZ4mTzcJ5VTw6wCt2xND KdRcvI5/1rm+uhIJ1oFD41y6F7a0xVb/z1Osjm/NMqBshFwDfePdKKh0LoczcGULsBmi MaIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oE7F/pmLVdPW6Kss7G1myKGx46vWPSP6E/UNtsRt16o=; b=G6VmBaUCsO8lnzqo6oXjoBQMJbN8ItEvqlhyiTQPnQv3tFgzWrY4nKqUY0l15BBbZ6 c0eNvMPg1XWBnBVEPtafZMfS0P40d3cTbKWMY04/g9DOuVHwMksiUW/RzSLD+toTsITy 0730ShpFfQs9mzKXV/D7fnfvp0vipjefffNIvY3RGsYOU6uN1NW6f4mrKHQk2MXi8urJ MrviEgkhWHKU1NaUHXzy1AURO7QIqg/l2rI9/oUgHnSElHy6MRinw+5fh1StotC+Yhmh Ul4aKhNg6uwE3PmLknyl5vawZexTx0CfwjcOR3No2SEz3vvjc5Q1XkPHzO1aP7Yyrs6L 5mFw== X-Gm-Message-State: AOAM530KdnRtG5K2T33EWRyZp7leLv9aUflPp/pB1W1ocdjGb6yi2LyU 2xRJLgv8L78d5qMaqWaygiCjvv/NcEwa7nI5DeQQMyD8 X-Google-Smtp-Source: ABdhPJyJI18q4B8OUxZuMvoS1mj8pxkxiDumBrCzNAdkvhGBtOcg87npEFUP90iZxeMRUalbYttsVUNj601PzypY/uw= X-Received: by 2002:a2e:9b0b:: with SMTP id u11mr27058337lji.463.1629720290890; Mon, 23 Aug 2021 05:04:50 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a2e:6f04:0:0:0:0:0 with HTTP; Mon, 23 Aug 2021 05:04:49 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Mon, 23 Aug 2021 14:04:49 +0200 Message-ID: Subject: Re: sysctl is too slow To: Alan Somers Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4GtWBy2xBJz58xV X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Gn3TkNwr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::22b as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [0.86 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22b:from]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_HAM_SHORT(-0.14)[-0.144]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N So is this something you plan on fixing? On 8/17/21, Alan Somers wrote: > Actually, I did get a flamegraph, and only 0.77% of samples were in ZFS. > > On Mon, Aug 16, 2021 at 7:19 PM Mateusz Guzik wrote: > >> On 8/16/21, Alan Somers wrote: >> > Yes, I see what you're talking about now. There are a bunch of linked >> > lists in sysctl_find_oid etc. Good point. >> > -Alan >> > >> >> You still want to get a flamegraph, chances are most of the problem is in >> zfs. >> >> > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik >> > wrote: >> > >> >> Last time I checked lookup of a sysctl was very bad with linear scans >> all >> >> over. >> >> >> >> Short of complete revamp of the entire thing I would start with >> >> replacing the scans with a RB tree at each level. As is if you indeed >> >> have 5000 datasets, you are doing increasingly longer walks. >> >> >> >> On 8/16/21, Alan Somers wrote: >> >> > ztop feels very sluggish on a server with 5000 ZFS datasets. Dtrace >> >> shows >> >> > that almost all of its time is spent in sys_sysctl. ktrace shows >> >> > that >> >> both >> >> > ztop and sysctl(8) call sys_sysctl a total of five times for each >> >> > sysctl >> >> > they care about: >> >> > >> >> > 1) To get the next oid >> >> > 2) To get the sysctl's name >> >> > 3) To get the oidfmt >> >> > 4) To get the size of the value >> >> > 5) To get the value itself. >> >> > >> >> > Each of these steps takes about equal time, and together all five >> >> > take >> >> > about 100us. If the time per call is mostly syscall overhead, then >> the >> >> > process could be sped up by 80% by combining all of these things >> >> > into >> a >> >> > single syscall: return the next oid, its name, its format, the size >> >> > of >> >> its >> >> > value, and optimistically the value itself, assuming the user passed >> >> > a >> >> > sufficiently large buffer. >> >> > >> >> > Am I missing something? Is there any other reason why sysctl is so >> >> > slow? >> >> > Or should I forget about it, and try to export ZFS's dataset stats >> >> through >> >> > devstat instead? >> >> > -Alan >> >> > >> >> >> >> >> >> -- >> >> Mateusz Guzik >> >> >> > >> >> >> -- >> Mateusz Guzik >> > -- Mateusz Guzik