From nobody Tue Aug 15 21:29:45 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RQPYR21Yqz4qRmv for ; Tue, 15 Aug 2023 21:29:47 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x29.google.com (mail-oa1-x29.google.com [IPv6:2001:4860:4864:20::29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RQPYQ4yCFz4Cn0 for ; Tue, 15 Aug 2023 21:29:46 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-1c4c5375329so2404018fac.2 for ; Tue, 15 Aug 2023 14:29:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692134985; x=1692739785; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Wcp/Wyu7R77A5GAps1LUbVJYfFVXP407Nudom8aSM0o=; b=cL7YERv2+10yMuSHBE/lFsAzw1YpNDLw3bvsKXm/wyLts00fJo+0iP/GzBYpx5Poxr NnHAxjPBuOrZiKmhdXSluXEKFQm5+uOnFBEqIcoRmNhqVoVypqhVqH1LKG0zGdC0T0jV cY1WeWarjWuxp/SJioPd4RGUcbc5bb6xoHAyATT5ChBXpHc4Y5fuJr/n2Mw3yn0tWlok I1Dt8QKUT32n3AUGCfPwJTeSo4xgMLbgbxP11MJS50/+XuyUtZxcMHHUwq4yNXFkDUAw LurMfpqHM+qSs/flhyFVrSjDQkW8GbngF/FHhK1QFLgVhVGH3OqpQPBlo/Vwnfsh1nP0 1pZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692134985; x=1692739785; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Wcp/Wyu7R77A5GAps1LUbVJYfFVXP407Nudom8aSM0o=; b=VwhMZplbzMU3mBI2vjNEPbEOMpNWQoBc504nwQIsSw0D76xaEYWXhMXgcpmgVYz5vd VO2AncAmqpc3lyXWP0GwQ0PMjPrXKT5rRcuVS5zsIKN78tXcEQpzDjnLRR217GYekuH8 d+hWuNtTfw8VgZoLBbC9dUFxW7CBjWDtQP6TGMCFboA6FlVOH2A0MhIuOlza1zl98n+8 gQgFUuonGhz1uqUjXxqCQNUlp+o+qQBEJtbkN5h4SyK9VboQ4MY556Ngc1XF7B6UiYX5 l+xAqbCXdpftB718andtk3WALFqw7vVFBSOjw8Ojp1lx+U+i4bW2BqIC4GpAS8h+p8fb dEsA== X-Gm-Message-State: AOJu0YxNKpwaxOhugenL5Gscb7oW5Nh9iur2TqKOi87Z4nbaZBTjgxdp UA5kDf50HO2HsT3eqI/YBoMajjZX4gpGlB3L0NkB07HP5zQ= X-Google-Smtp-Source: AGHT+IHakXZUHNnSL1llhO9+XrlTJQm8T8eHlsGf6QQaERcuDwvP+z3vK09hKLCH/q58DJpgzTPnebY/D9uDSC1HZUE= X-Received: by 2002:a05:6870:e990:b0:1bf:ce5b:47a with SMTP id r16-20020a056870e99000b001bfce5b047amr13746451oao.52.1692134985598; Tue, 15 Aug 2023 14:29:45 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:745a:0:b0:4f0:1250:dd51 with HTTP; Tue, 15 Aug 2023 14:29:45 -0700 (PDT) In-Reply-To: References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> From: Mateusz Guzik Date: Tue, 15 Aug 2023 23:29:45 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RQPYQ4yCFz4Cn0 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] On 8/15/23, Alexander Leidinger wrote: > Am 2023-08-15 14:41, schrieb Mateusz Guzik: > >> With this in mind can you provide: sysctl kern.maxvnodes >> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes >> vfs.recycles_free vfs.recycles > > After a reboot: > kern.maxvnodes: 10485760 > vfs.wantfreevnodes: 2621440 > vfs.freevnodes: 24696 > vfs.vnodes_created: 1658162 > vfs.numvnodes: 173937 > vfs.recycles_free: 0 > vfs.recycles: 0 > >> Meanwhile if there is tons of recycles, you can damage control by >> bumping kern.maxvnodes. > > Looks like there are not much free directly after the reboot. I will > check the values tomorrow after the periodic run again and maybe > increase by 10 or 100 so see if it makes a difference. > >> If this is not the problem you can use dtrace to figure it out. > > dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or > something else? > I mean checking where find is spending time instead of speculating. There is no productized way to do it so to speak, but the following crapper should be good enough: #pragma D option dynvarsize=32m profile:::profile-997 /execname == "find"/ { @oncpu[stack(), "oncpu"] = count(); } /* * The p_flag & 0x4 test filters out kernel threads. */ sched:::off-cpu /execname == "find"/ { self->ts = timestamp; } sched:::on-cpu /self->ts/ { @offcpu[stack(30), "offcpu"] = sum(timestamp - self->ts); self->ts = 0; } dtrace:::END { normalize(@offcpu, 1000000); printa("%k\n%s\n%@d\n\n", @offcpu); printa("%k\n%s\n%@d\n\n", @oncpu); } just leave it running as: dtrace -s script.d -o output kill it after periodic finishes. it blindly assumes there will be no other processes named "find" messing around. > Bye, > Alexander. > > -- > http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF > -- Mateusz Guzik