From nobody Sun Oct 16 14:15:51 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr2Gr6Kl2z4fK15 for ; Sun, 16 Oct 2022 14:16:04 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com [209.85.217.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr2Gq66l6z3QVj for ; Sun, 16 Oct 2022 14:16:03 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-vs1-f53.google.com with SMTP id 3so9228694vsh.5 for ; Sun, 16 Oct 2022 07:16:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YTxtxCB7Sb7tHfxXF6I2GbQKRViHVLdOLAJUuG/d5xU=; b=RDJD08Rkx90KJkKotTZ63K9J4lgjyIdg67gEkOWkQ04tNwpCLe+CBs8cwxF+Md7GWY jm62ahSepCTYjU1EKdDd+kn11Yw7fcyQdUxy4dHJ+RzJpL0I0Y8JG/4ajjvEZMW19jxk AXYXarzXdONqd7W/VKncZOOjA6dpZqKnLUlPa0eRyexL/uJxeYUQmrohoXgD2FarRV/P CzVYYqKgWVlRZAw9UtxVrfz8MoJ4oP8ES0O0JUBSP8p6e9SXgvpIUcOtskhHu8wxDF1O csrEi1CicLsRQzHPSvdMFm9M/0Wt2MsgPWq5C5IgVk7l6tBJSsFGnSQc9h6xteZ+0dgT BJtA== X-Gm-Message-State: ACrzQf0EwjRZvz+pVdtXxmI2rojTmiubUGpOVWq/8IuXDtMlkxJJRUA5 QxsqlIfFYV058LLwWAghYzerQYxvr98tR1XUcpaTHnc3 X-Google-Smtp-Source: AMsMyM6fHlJKRsfN2xK1+oRXAb/TXM/KFf605UZULMrueqrS+qcDrtCwxR5QesGy2s9GZUwKtC6dgKMKGa1NAvUpwAE= X-Received: by 2002:a67:b303:0:b0:3a9:61e4:2769 with SMTP id a3-20020a67b303000000b003a961e42769mr1750076vsm.0.1665929763098; Sun, 16 Oct 2022 07:16:03 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Sun, 16 Oct 2022 08:15:51 -0600 Message-ID: Subject: Re: zfs with operations like rm -rf takes a very long time recently To: void Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Mr2Gq66l6z3QVj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.53 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_TO(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.53:from]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[asomers]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.53:from]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N The usual reason why rm gets slow is because your pool is nearly full and there's a snapshot. A snapshot means that rm doesn't actually free space; it just rewrites metadata, which requires even more space. And when a zpool is nearly full, writes always slow way down. But if that's not it, then you should also check gstat to see if the disk itself is slow. On Sun, Oct 16, 2022 at 7:57 AM void wrote: > > Has anything recently changed in -current that would make file operations > on zfs such as rm -rf *.* very slow? > > What would I look for and how would I test it? > > system is FreeBSD 14.0-CURRENT #5 main-n258595-226e41467ee1 on arm64.aarch64 > using GENERIC-NODEBUG kernel. > > the zfs is zroot on usb3 on a raspberry pi4 8GB. there appears to be plenty > of resources. cpu speed is 2.1GHz. zroot is external usb3 hd. > > Right now it's rm -rf-ing /var/cache/ccache/* which is 5GB max and it's taken > over 10 mins. It was never this slow. No errors in /var/log/messages and none > yet in smartd. zpool scrub last ran successfully 3 days ago. > > last pid: 4324; load averages: 0.17, 0.10, 0.12 up 0+02:10:34 14:40:55 > 77 processes: 1 running, 76 sleeping > CPU: 1.6% user, 0.0% nice, 1.9% system, 0.2% interrupt, 96.4% idle > Mem: 550M Active, 803M Inact, 2224M Wired, 40K Buf, 4239M Free > ARC: 1293M Total, 381M MFU, 725M MRU, 1124K Anon, 30M Header, 156M Other > 938M Compressed, 1906M Uncompressed, 2.03:1 Ratio > Swap: 16G Total, 16G Free > Process id to show (+ for all): > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 3871 root 1 20 0 12M 3648K zio->i 2 0:10 0.39% rm > 353 _pflogd 1 20 0 13M 2108K bpf 1 0:00 0.00% pflogd > 1441 mailnull 1 28 0 25M 9508K select 3 0:01 0.00% exim > > TIA, >