From nobody Sun Oct 16 15:42:00 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr4B92NxRz4fTP0 for ; Sun, 16 Oct 2022 15:42:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-19.consmr.mail.gq1.yahoo.com (sonic313-19.consmr.mail.gq1.yahoo.com [98.137.65.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr4B83cM2z3byf for ; Sun, 16 Oct 2022 15:42:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1665934926; bh=qhfps6j0ExpOd3xi13PGqfi7ImJ+ZZfwVPK50k8I0jQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=e4KJoZzN54T///glVvO9/NOBwaY55edWS/r3ITU1XvrbaAIAUZGv4lQUQ+/zKxss2jPGDsBdKKhZiC5/4aZdO7NJbE0b3w2mY3iUREwZ0CBx0bDrt1ZphivZ88kZOYC9D0ob5IwQ1KwPZlYpgUc0VjGiq5e3vFCuZhItedIuQKFIZCBu/JFp+OnfMPdcD3VHB68s1EEKJQrC7ogLerZdtJ3+Kb0PsKZ66Go2ok/vHFB67hiSnS0Z3ad8sRwzr/iUpSuTv1m92+YTtbHIyqEWw+Cz63In+NkUXXQm6wGwm3IAakFQwlU9j4jokl4BxgO2RYJ4VUNvc1AhnKzF9425Xg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1665934926; bh=Cme1lBSSlrfDg5zgG4cpZS+D6iJ9UQuUuKcVv5V/vel=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=i21AVBo7IANsA9458RfIR28p68Hwbj41VoG/XunRMSmnLGaQT6Wne6LqGCh0BNQKFAwXNl+KvLTmjjNlOa3cnaiserwgxoatj+H++QkQlcXQvJ34QY/0NW1LIxj4K1BM/dXOUDzy15dW3XZ3z2fup4JEy4tm3lNXFcNvZgf4tUGxteghWaiGmDto4+ZoaRyCYyJtx2s+pq3DPkLEsyDcfGRIY6mK+Rkb98zabD0wILNBaw2eAXlzfJGaUntFiCEVp6E5x2Bp+AYQyRFvOKXxxGFmWPjS6JTI7QhKecAkZmo1mEnsrIZadERFiu6jmscem+WDNrD1lWHLpyjXqlOSaQ== X-YMail-OSG: VDjQH3cVM1nb0I3anso5ke2i6x9MEgpRusZU_W3nMmLOmzSat0OIvenZwX3kDNt vkH08DO_Cg7pp6cWqjwa7ixN9K0jjgRbYP6gNanYcQBlwvRQeSjA2vI1qGWGavfq1sin7atS6uuE sVsVk4d2.5o3zilgkcr6TztajQVMr2B54rOjL_6uAiE69Co38ajhI24qfvTS9fCpWE_pjvSUAPqx iFufyWfSfwqW_IDZhHfKbSNcPMj5po6kmcRvUxgcLYSlk8Hzhayf_tD_Ybym8uQ_zPGqeLyNAghu MHgCsPY0R6WNRJH_Px5Zsr6Ww0a3o0QUXuLDltQEPkU7jI9EljP_rrxARccrmv7R.0zbOUoCWMJO TFyFLbYiw4QsGXUvJG.8UXDVkEH4sF_KszoBlJMIAVh0wGEYsW2vOWZXKvyLF6JKB_RJYay_dF4X H5YUptHpsoVyTXjR1c9Ckf4Z4vI5mETZZ6EN9HRh08pO4IQ1NLLq0ULzHtHM_YHs9qdpFrzRJuNv sPpL1QFiahDvsqQ.nStqxz9ENT3anas5nSoMct3bJxANQZFR0wIC1CTc6ZiwGj70R0AL.KESBYNj y1SFo3aO2C_6fzaJANPW52UiQWQcT558qBCr26OsZ9kdDOYQTfa1omBUFq0HtR5v7Z4d9v0ah5A2 _8CtdUatf4NOTYkYzXAd1dvUhdDNP80j6XpfFuLaQLL6xzIn45luG.IyvJzTitUOKNz9KLOIgi2G N8T6hXxYQuoT1g8wrz2MjWwsXD4dT_Y04NatB4UHE2hnU4A1SxApl5pnvy2mJ_DES4Pyv7xULKZ0 Jo.mcNYU9YUXuaz6tWatxqgXqHIX.YuPPDS9zG_0AW1aVAWCKFadcshyxu84T1NA4_mn4_fhYI0t T9So5kwuFIl_WbHiy3VliFM7_9n5iYUsuPPeargoKkdHpPrCBXNfsC7hsiQH3LbPA1bN4XYg6fBn OQJORZhtV2Gd6qPtNzjcWZmuj9he6CA57xAjXXyOkyXenGkcK7r8TWeqD84QSxE18TqalHVeDTdM XwPhVPFWOWd9eodPE.AZr3IYv5u9Th25gxGPwUHJ6exAnJY9XyaK7ERC4R7UTKLy1UjPYZ0rHOTh .jcaWeCjIDy8OtADB16qBNEa66iQJoKMq9D5XmX_I0V1e0j5rKE8kc2Tutc2mLhBIdZGZETaK6F3 _M1rgq2T5CZUKc4GMthksv.YJRz.eb_6fdx7athc7JSsunpkOUZYGCGECvJHmj.ce3EGLNiPfQWR lSpqtsHGx2BB6ism7fdWugQHShc81cADntuN..k_CZs_.dEoiOyY6A1JW4.ZYQLI6dXaQ33ebRl_ 0JsM.X2FKnQRxfRbCa9KQtFTqsdLSh0eoa1X4qzhh_6toDrKr5oYgDREE3TDLDerneAKqbjMwHu1 YzDIQRwVBpzVTdonL2gNDQCQZ8okN4lIObMhHSYrlSrAMQcbCTNHeajSi9JCdyFpajHTPhtOhHg. iB1ite_JZuTU8CplNxv4iiv_OIUjtgflXKTNrDsKITP4mMl03Sa6EjhFW_0HdmcUx7oVPq9akGHy N8yVbggWg3o1RFcmVIwJ._rrVrKIqag9jNQu_7zT1R8z5G0LUF4k0YcVgm94B9bRNoaeq.Tb9Xnh PcWg4KH53T4LE9yV72bHV9uPRPSU0k0zYpR23IhIOSngSmfRFAa5hJua0te4NaDkJNP1sQ6GLhR2 RVhc1WYz_zMy7lTvH20YJAryoBiPMa5DhvFNzrtyKhDJVVpBPDw8Hm_gDau7K4QGXiYXoowY5hvO MGV7z4YpgGKgueZqE6poKTTsExvUd.4pT.BgfouzMUnxrilepY1PwvKQwyA7Gg5bN8mk6lShIRVQ mqvhdI39XH_2UeMGeJ8KHw92MY7BEoIBm5HsvAZ87ZBWWyj2_AlSsfHC97FJA21Ejcmm1.7DV4w3 Rdx1W_6JUxCDcbjDrIBpyVcJVjS_K1TJWv2DOFdLd06qXVbsvdBCanp0ncJIkeXgCfYx0y2eBaBC gdfCn5M5rOYWar1ClkvgmjZmx6FJPyvVQjLik0DpNk3oBEIhYV0.DG27ennKg6Xz5cGNM2WCRwN7 rWwWAy_pycl8FULXoLaNp6Tl6Vi3kQqtMFl7bYqQbrEqe9jqToC7_OJeI_GKftwZey4STNUIKhOv 4HcsyJ9NVeUTLHhb.UQwpeB8xDFBkpl2B3G7I4FAOkfm5R3Ia_N9upuL1bys5JEnErjd0AmuAI_q oIJq65f7Tg1r1sLg6.xVpA3GAqbHSC.v_zsVNuaaQz9._9SHg48hbbZpdL5QhExSsccg- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sun, 16 Oct 2022 15:42:06 +0000 Received: by hermes--production-bf1-585bd66ffc-gfghw (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3bd51f861688a041c9e15a52c054df13; Sun, 16 Oct 2022 15:42:02 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: RE: zfs with operations like rm -rf takes a very long time recently Message-Id: Date: Sun, 16 Oct 2022 08:42:00 -0700 To: void , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3696.120.41.1.1) References: X-Rspamd-Queue-Id: 4Mr4B83cM2z3byf X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=e4KJoZzN; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.82 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.82:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N void wrote on Date: Sun, 16 Oct 2022 13:57:04 UTC : > Has anything recently changed in -current that would make file = operations=20 > on zfs such as rm -rf *.* very slow? >=20 > What would I look for and how would I test it? >=20 > system is FreeBSD 14.0-CURRENT #5 main-n258595-226e41467ee1 on = arm64.aarch64 > using GENERIC-NODEBUG kernel. >=20 > the zfs is zroot on usb3 on a raspberry pi4 8GB. there appears to be = plenty=20 > of resources. cpu speed is 2.1GHz. zroot is external usb3 hd. >=20 > Right now it's rm -rf-ing /var/cache/ccache/* which is 5GB max and = it's taken=20 > over 10 mins. It was never this slow. No errors in /var/log/messages = and none=20 > yet in smartd. zpool scrub last ran successfully 3 days ago. >=20 > last pid: 4324; load averages: 0.17, 0.10, 0.12 = up 0+02:10:34 14:40:55 > 77 processes: 1 running, 76 sleeping > CPU: 1.6% user, 0.0% nice, 1.9% system, 0.2% interrupt, 96.4% idle > Mem: 550M Active, 803M Inact, 2224M Wired, 40K Buf, 4239M Free > ARC: 1293M Total, 381M MFU, 725M MRU, 1124K Anon, 30M Header, 156M = Other > 938M Compressed, 1906M Uncompressed, 2.03:1 Ratio > Swap: 16G Total, 16G Free > Process id to show (+ for all):=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND > 3871 root 1 20 0 12M 3648K zio->i 2 0:10 0.39% = rm > 353 _pflogd 1 20 0 13M 2108K bpf 1 0:00 0.00% = pflogd > 1441 mailnull 1 28 0 25M 9508K select 3 0:01 0.00% = exim "The Design and Implementation of the FreeBSD Operating System" 2nd Ed. says about ZFS (page 548): "Like all non-overwriting filesystems, ZFS operates best when at least a quarter of its disk pool is free. Write throughout becomes poor when the pool gets too full. By contrast, UFS can run well to 95 percent full and acceptably to 99 percent full." And page 549 says: "ZFS was designed to manage and operate enormous filesystems easily, which it does well. Its design assumed that it would have many fast 64-bit CPUs with large amounts of memory to support these enormous filesystems. When these resources are available, it works really well. However, it is not designed for or well suited to run on resource-constrained system using 32-bit CPUs with less than 8 Gbyte of memory and one small, nearly-full disk, which is typical of many embedded systems." (Note the full-disk part and 8 GiByte being at the low end of the RAM size range.) Page 523 says: "ZFS takes advantage of the abundant processor power available with current multi-core CPUs. Because they are much faster than storage, ZFS can afford to checksum everything." The book is not explicit about RAM subsystem performance tradeoffs for ZFS. One property of the RPi4B's is that they have very small RAM caches and one core can saturate the memory subsystem if the RAM caches are being fairly ineffective overall. In such contexts, multi-core need not cut the time things take. (But I've no clue how likely such conditions would be for your context.) A cache-busting access pattern over much more than 1 MiByte memory range drops the RPi4B performance greatly compared to such an access pattern fitting in a 1 MiByte or smaller range --no matter if it is 1 core or more cores that is/are trying to be active. Independent of all that, something like: # gstat -spod would likely be interesting to monitor at during a time-taking "rm -fr" . I do not know if the "rm -fr" is deleting a lot of files that also have unchanged content in a snapshot. Such files are not actually deleted. The information about where the file should be visible is adjusted instead, leaving the snapshot copy(s) available for access. Such adds to the disk space usage by writing more metadata without deleting the snapshot related data. =3D=3D=3D Mark Millard marklmi at yahoo.com