From nobody Sun Oct 16 19:55:21 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr9pN3W8dz4fxg1 for ; Sun, 16 Oct 2022 19:55:24 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr9pM49vpz47lW for ; Sun, 16 Oct 2022 19:55:23 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-1321a1e94b3so11410799fac.1 for ; Sun, 16 Oct 2022 12:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HwzsCQr91gQYkeUNrzxCra7gg1FZzmXEPqjZ9lJIcOc=; b=E7N+HPuI5S80TGA5/FWluRw4f2EzKMYhHssUS9+fpbplzDYv0sZ9ClNx1BHU8cZCkh T8QmxKZXsSoF25MKJNjzfVHyeqILOjj4vBqe2812FrqjHxNlH5pxiLm7RAdW2aPGFkOI hkynoCUezRsYO0uSMJvcWBnmZ77nuRxv0kXTjkOQbLbZMvcAQpoVBN2uMj4N9d/HE93Y a8jQyJSfvdrsqXc85A3iEvj5211sLwxxQO2lWJ74Llqxgz98DCSADSjujYFhVC1FiZzN J612nwOFy0TN+U1vjd0oM7WM1xYlMA/UnbYqrJsmyW4RR9BkthbycF+YSNcDsP5LWXl0 ZGug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HwzsCQr91gQYkeUNrzxCra7gg1FZzmXEPqjZ9lJIcOc=; b=AWEhH6qHfP0KLJMiYGona5P1vGaDKeaSQslN0FvhOrb1p+K+TrA8MQR46g/qLlUiqR Gi1VpI5zwK+Ax6iIM5fm1EL8FUAl4qy+l1ls11oGArXs0l4I6P/loO6d4/8OXEMCoMsD 5mLiOQCpeyVIxWL9SfQtFTVIjjZxTQbP7BYFKbek1NLt01t8AIrDuohcr7YkdZRMK//r vfaMrZPxV3P4LGH+dZjq1xSOBjKX6DTvJkfVamZHRZHzXyMHHtF8Yt3cClsSummJdSwA sseqFWMYZIzLXY7nT9+LOYITmnwbzSox9nwuwbALKtDB35JloW9dbrzH08RmZhzLBWBE e87A== X-Gm-Message-State: ACrzQf1sCwUw8gmzKWqXKZe0OJ8NRWFNx7MRo6JeF77DnwFGINdcRaIo 1pA6ebS7EJjr8pfzmpDUtz5t59rj+27lawpcFsk= X-Google-Smtp-Source: AMsMyM6he4+GGWP9n3Lv9Q/gQbUSpO1NChIQNunixcnytiWUjzMvrtC0jW+Wzciq7zkui+SH5QYnrMj44a0QNBOjPXI= X-Received: by 2002:a05:6870:a7a4:b0:136:7c39:979e with SMTP id x36-20020a056870a7a400b001367c39979emr4174106oao.96.1665950121894; Sun, 16 Oct 2022 12:55:21 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:351:0:b0:478:5e54:a9c8 with HTTP; Sun, 16 Oct 2022 12:55:21 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Sun, 16 Oct 2022 21:55:21 +0200 Message-ID: Subject: Re: zfs with operations like rm -rf takes a very long time recently To: void Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Mr9pM49vpz47lW X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=E7N+HPuI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::33 as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::33:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_TO(0.00)[f-m.fm]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N On 10/16/22, void wrote: > Has anything recently changed in -current that would make file operations > on zfs such as rm -rf *.* very slow? > > What would I look for and how would I test it? > > system is FreeBSD 14.0-CURRENT #5 main-n258595-226e41467ee1 on > arm64.aarch64 > using GENERIC-NODEBUG kernel. > > the zfs is zroot on usb3 on a raspberry pi4 8GB. there appears to be plenty > > of resources. cpu speed is 2.1GHz. zroot is external usb3 hd. > > Right now it's rm -rf-ing /var/cache/ccache/* which is 5GB max and it's > taken > over 10 mins. It was never this slow. No errors in /var/log/messages and > none > yet in smartd. zpool scrub last ran successfully 3 days ago. > > last pid: 4324; load averages: 0.17, 0.10, 0.12 > up 0+02:10:34 14:40:55 > 77 processes: 1 running, 76 sleeping > CPU: 1.6% user, 0.0% nice, 1.9% system, 0.2% interrupt, 96.4% idle > Mem: 550M Active, 803M Inact, 2224M Wired, 40K Buf, 4239M Free > ARC: 1293M Total, 381M MFU, 725M MRU, 1124K Anon, 30M Header, 156M Other > 938M Compressed, 1906M Uncompressed, 2.03:1 Ratio > Swap: 16G Total, 16G Free > Process id to show (+ for all): > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 3871 root 1 20 0 12M 3648K zio->i 2 0:10 0.39% rm > 353 _pflogd 1 20 0 13M 2108K bpf 1 0:00 0.00% > pflogd > 1441 mailnull 1 28 0 25M 9508K select 3 0:01 0.00% exim > Just in case, can you test with this: sysctl vfs.zfs.per_txg_dirty_frees_percent=5 As in, can you repeat the test multiple times, flipping vfs.zfs.per_txg_dirty_frees_percent between 5 and 30. -- Mateusz Guzik