From nobody Sun Oct 16 13:57:04 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr1sn4g7Nz4fHTm for ; Sun, 16 Oct 2022 13:57:49 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr1sl5vZbz3PNs for ; Sun, 16 Oct 2022 13:57:47 +0000 (UTC) (envelope-from void@f-m.fm) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 4EC7132007BE for ; Sun, 16 Oct 2022 09:57:45 -0400 (EDT) Received: from imap46 ([10.202.2.96]) by compute3.internal (MEProxy); Sun, 16 Oct 2022 09:57:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm3; t= 1665928664; x=1666015064; bh=SsVzU2rz9lAh1Cm0Tdf07ZyPKQalpappM2q J+QOxaIo=; b=BHrYAKg4xQ2DkMosebaP/OK0ptlTJBFpFNgwo2X5RdWoon+7cnm DxVNrc3FpyBX3KGNREyMDMS0dTExxXL48IuXyelAelBH8gTgGsqTBawDAA2QMWdL BcNDgnJo7/FO78X6rsgJnLBttmnqU9uAkSOZrob0h8vgB1M0VhEsMqoy/9NcHp3N No+cjuRjyazcQvwptPgrBfDhKmZUVg3kRxx621y7LG593fZl2QDYFEecAtroZk2v ZJmLuobTtrMcFgTqiYHwWp10VwCeLGuQ8yDWhfkXogzvexyLHRNr5lFZxL9N8PN8 nbQPJFvKfn26DLZTI+7W+cu92oCUSPbf3jg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1665928664; x= 1666015064; bh=SsVzU2rz9lAh1Cm0Tdf07ZyPKQalpappM2qJ+QOxaIo=; b=I kYg7yVSGMsqkrVrud4FirrimykhFSRg89SsCjPfQ2aQ7qyIj9eU01EcJE0dwJ8/2 Iht1laDzxZo8PdxuXp+hdwJ14jv9meF5mV3E4vfEOOCtymEo/dSV2ZfIKEpP4338 NJ7VBhV3LHRLhmQdyJyM2fvH6hNcNLoz185k2bPhXl3qgLxIQzGwBcJUZ+iHuntS ESHCU+h5qoeBBszo+HtTgIPxevtOZU4dBfj5tsR+Z2XmqtlhB25Adly4WefK0ApM RIzXr0o905Lxc7rQ3Zgja1s6CJ5YVoDrVh5gSrJHPXvQGpHOU79nXkBWh0WJXktZ zZwSbex9eO0ZgvfOC9afg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfeekjedgjeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsehttdertd erredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrght thgvrhhnpeeftdfhhefgjeeiteevfeeghedvkeefgeetudffffehfffgjeehtedukefhje dtveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehv ohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 8D9C42A20079; Sun, 16 Oct 2022 09:57:44 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-1047-g9e4af4ada4-fm-20221005.001-g9e4af4ad List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 Message-Id: Date: Sun, 16 Oct 2022 13:57:04 +0000 From: void To: freebsd-fs@freebsd.org Subject: zfs with operations like rm -rf takes a very long time recently Content-Type: text/plain X-Rspamd-Queue-Id: 4Mr1sl5vZbz3PNs X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=BHrYAKg4; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="I kYg7yV"; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm X-Spamd-Result: default: False [-4.59 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N Has anything recently changed in -current that would make file operations on zfs such as rm -rf *.* very slow? What would I look for and how would I test it? system is FreeBSD 14.0-CURRENT #5 main-n258595-226e41467ee1 on arm64.aarch64 using GENERIC-NODEBUG kernel. the zfs is zroot on usb3 on a raspberry pi4 8GB. there appears to be plenty of resources. cpu speed is 2.1GHz. zroot is external usb3 hd. Right now it's rm -rf-ing /var/cache/ccache/* which is 5GB max and it's taken over 10 mins. It was never this slow. No errors in /var/log/messages and none yet in smartd. zpool scrub last ran successfully 3 days ago. last pid: 4324; load averages: 0.17, 0.10, 0.12 up 0+02:10:34 14:40:55 77 processes: 1 running, 76 sleeping CPU: 1.6% user, 0.0% nice, 1.9% system, 0.2% interrupt, 96.4% idle Mem: 550M Active, 803M Inact, 2224M Wired, 40K Buf, 4239M Free ARC: 1293M Total, 381M MFU, 725M MRU, 1124K Anon, 30M Header, 156M Other 938M Compressed, 1906M Uncompressed, 2.03:1 Ratio Swap: 16G Total, 16G Free Process id to show (+ for all): PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 3871 root 1 20 0 12M 3648K zio->i 2 0:10 0.39% rm 353 _pflogd 1 20 0 13M 2108K bpf 1 0:00 0.00% pflogd 1441 mailnull 1 28 0 25M 9508K select 3 0:01 0.00% exim TIA,