From nobody Fri Aug 18 13:29:30 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RS2lw4bPKz4qXFB for ; Fri, 18 Aug 2023 13:29:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc34.google.com (mail-oo1-xc34.google.com [IPv6:2607:f8b0:4864:20::c34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RS2lw2yBJz4XPv for ; Fri, 18 Aug 2023 13:29:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc34.google.com with SMTP id 006d021491bc7-56ce1bd7fc4so632768eaf.2 for ; Fri, 18 Aug 2023 06:29:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692365371; x=1692970171; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=VGYKlKvPUQpnLnfgcw4jCb7cGknb3XnaaP3tbyTid/w=; b=rHBjDL6RdG1lL/VmOXB9D5W21nlt2OCJBVee1vQuNmBrp8f3m+BydKNVWBrHr1OpeZ vXzBcs17YhEQi0W7IvpVtstHKOsqx8Pt4C5ofgaaXWa62BhOBmCi9BPwlexc+aCw10bZ rEn+oNglHxf9BANZ9oaZUmax5/ioSXXTA6R0zvZy3NVmU1rPBlvbDMbdsbvKHwIJltUj Pk7uftyaH6vqQTgHhly9uDMQe+48Vsptm9C1FEAUqkUbJvv8GEerBuJeeaHpUTbALiFB wX/QRQ1XGE7Br5FZeMrkA+P8uOOzwBuGQrdO/FmoAv6F1kXbF0d5M5ZqwxtJkXxHKfnJ ktaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692365371; x=1692970171; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VGYKlKvPUQpnLnfgcw4jCb7cGknb3XnaaP3tbyTid/w=; b=SN6WRiIdkrvPv/EIU5bIQ2HLhRvkIwWypSpNP3YoyK7SNU72C+KHqHZWuOMrMVconL SxA/bBX+QNkR6zc2nZy8kdbJgTlfWzHB5im++saYYAQHymhR9w9tA50691Wt0papmA98 V5ER9sq7HTGtS8cPnOulaYfEs9KIWC+QDaeNxVOd2fBSgYMemPvU1zOkXx92MQeM0ytf KAv4PtKfXn99o7KJrdNZ9Nw9UFXi9nLMQOrbhz6i87EmnBdi03GQxmS7d4ti/1bsvHwL c2un6VEamHVmM2/W5L9EXy9zclVLPXYFny8eL9gp59vHZn8MYZFfnSts1ThjNz+uu+Tx vlsQ== X-Gm-Message-State: AOJu0Yy16iqz2quA+BGovZJFx8uIdZMs2nO13hdcB03AxxAw5mWCw24y k127zkG0M6G76aO6ZGDepNWgcTTPZTo89qO9lESQT4s2 X-Google-Smtp-Source: AGHT+IEZ5M//6ld/POlsKGaRfFFwo4AjhSREjIWckd0AsWFGu9I/YHl+qB4bsybSYF+0Voas/ZFi2+NUFzkUfrW91bI= X-Received: by 2002:a05:6870:e893:b0:1b0:408a:1d14 with SMTP id q19-20020a056870e89300b001b0408a1d14mr2996673oan.44.1692365370865; Fri, 18 Aug 2023 06:29:30 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:79d9:0:b0:4f0:1250:dd51 with HTTP; Fri, 18 Aug 2023 06:29:30 -0700 (PDT) In-Reply-To: <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> From: Mateusz Guzik Date: Fri, 18 Aug 2023 15:29:30 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RS2lw2yBJz4XPv X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/18/23, Alexander Leidinger wrote: > Am 2023-08-16 18:48, schrieb Alexander Leidinger: >> Am 2023-08-15 23:29, schrieb Mateusz Guzik: >>> On 8/15/23, Alexander Leidinger wrote: >>>> Am 2023-08-15 14:41, schrieb Mateusz Guzik: >>>> >>>>> With this in mind can you provide: sysctl kern.maxvnodes >>>>> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes >>>>> vfs.recycles_free vfs.recycles >>>> >>>> After a reboot: >>>> kern.maxvnodes: 10485760 >>>> vfs.wantfreevnodes: 2621440 >>>> vfs.freevnodes: 24696 >>>> vfs.vnodes_created: 1658162 >>>> vfs.numvnodes: 173937 >>>> vfs.recycles_free: 0 >>>> vfs.recycles: 0 >> >> New values after one rund of periodic: >> kern.maxvnodes: 10485760 >> vfs.wantfreevnodes: 2621440 >> vfs.freevnodes: 356202 >> vfs.vnodes_created: 427696288 >> vfs.numvnodes: 532620 >> vfs.recycles_free: 20213257 >> vfs.recycles: 0 > > And after the second round which only took 7h this night: > kern.maxvnodes: 10485760 > vfs.wantfreevnodes: 2621440 > vfs.freevnodes: 3071754 > vfs.vnodes_created: 1275963316 > vfs.numvnodes: 3414906 > vfs.recycles_free: 58411371 > vfs.recycles: 0 > so your setup has a vastly higher number of vnodes to inspect than the number of vnodes it allows to exist at the same time, which further suggests it easily may be about that msleep. >>>>> Meanwhile if there is tons of recycles, you can damage control by >>>>> bumping kern.maxvnodes. >> >> What's the difference between recycles and recycles_free? Does the >> above count as bumping the maxvnodes? > > ^^^^^ > "free" vnodes are just hanging around and can be directly whacked, the others are used but *maybe* freeable (say a directory with a bunch of vnodes already established). >>>> Looks like there are not much free directly after the reboot. I will >>>> check the values tomorrow after the periodic run again and maybe >>>> increase by 10 or 100 so see if it makes a difference. >>>> >>>>> If this is not the problem you can use dtrace to figure it out. >>>> >>>> dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or >>>> something else? >>>> >>> >>> I mean checking where find is spending time instead of speculating. >>> >>> There is no productized way to do it so to speak, but the following >>> crapper should be good enough: >> [script] >> >> I will let it run this night. > > I have a 51MB text file, compressed to about 1MB. Are you interested to > get it? > Yea, put it on freefall for example. or feed it directly to flamegraph: cat file | ./stackcollapse.pl | ./flamegraph.pl > out.svg see this repo https://github.com/brendangregg/FlameGraph.git -- Mateusz Guzik