From nobody Sun Oct 16 15:12:15 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr3Xb0WY6z4fQfS for ; Sun, 16 Oct 2022 15:13:03 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr3XZ1JXwz3WRl for ; Sun, 16 Oct 2022 15:13:02 +0000 (UTC) (envelope-from void@f-m.fm) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 0487E3200319 for ; Sun, 16 Oct 2022 11:13:00 -0400 (EDT) Received: from imap46 ([10.202.2.96]) by compute3.internal (MEProxy); Sun, 16 Oct 2022 11:13:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; t=1665933180; x=1666019580; bh=BiIcRGi19T rEwgunu3kD7koVZ3TWWKJyzfSByQNokhw=; b=XpJJyLfj7o9HaSxJERmjcZsF9h rOY0LUvX2Fr2vFs+a9spEURNt6XXQk6+S1d87Yz3oko4UXM9uSU2A0Bj8WFHB2KZ cXoZ63ofDyjXacfvXejSV6oWKZca2SJGqYXpy1D9Uk4+OEKCCrZaY9bVkCzmLemO bKRrXys4xBd+nnlaJkHPCX8Dl8OlYuuKsLv5kcwul+3OBuYQ8M+obmaj7CYMAWsv /TCq0D9GefrT08ofjBlRRwa9/c+0mqQXn+8FaRinyjxIaJy3CE3XhdcnvAv2Qznz kG04t4aZwNoYyk/OhAKlPrT1KhG1oh+PvRSlZgSukfnHn+VhZHqLxgTcJhPw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1665933180; x=1666019580; bh=BiIcRGi19TrEwgunu3kD7koVZ3TW WKJyzfSByQNokhw=; b=B6kPz/dbinEbomSFXgG+8oSR75bEJjNjtpe1O/mh7sit NQlSPvPvlKmKW+jfxIunNb3YDjKLLKKHLNj4XCB6UgskdQB3niR9RMigYhAKD0sf mOX1sHH2fQPwEhsRzAkYb0sl0ceLkOHW6cx3y11P/QBbsVkwfzvwrGZiNQeJg//T NbIUAOsMq9yato1+xVXEuOns5gzZlkaghsEVooukgHXCuYbYwNVS84uQPI4oChWH 43YBH9zof8A2Hv+GUH/krEQ1Fx1SJWL7ICOkiOGGk2e/o4YKfWBR3SCemKrKCZMA 9qYFc9X0Or23RnOwuWsUXrGPAkANT2GJpSeTD3HFpg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrfeekjedgkeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeefgfeufefhgfdvfeegffetvdejudevgeegteeitdehhffftdejjeegud ffgeekueenucffohhmrghinhepshhkihhnfhhlihhnthdrtghordhukhenucevlhhushht vghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfh hm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 6C1912A20079; Sun, 16 Oct 2022 11:13:00 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-1047-g9e4af4ada4-fm-20221005.001-g9e4af4ad List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 Message-Id: <1625dc7b-81c4-4350-8f86-1b65f5a860d9@app.fastmail.com> In-Reply-To: References: Date: Sun, 16 Oct 2022 15:12:15 +0000 From: void To: freebsd-fs Subject: Re: zfs with operations like rm -rf takes a very long time recently Content-Type: text/plain X-Rspamd-Queue-Id: 4Mr3XZ1JXwz3WRl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=XpJJyLfj; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="B6kPz/db"; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.21 as permitted sender) smtp.mailfrom=void@f-m.fm X-Spamd-Result: default: False [-4.59 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N On Sun, 16 Oct 2022, at 14:15, Alan Somers wrote: > The usual reason why rm gets slow is because your pool is nearly full > and there's a snapshot. A snapshot means that rm doesn't actually > free space; it just rewrites metadata, which requires even more space. > And when a zpool is nearly full, writes always slow way down. > But if that's not it, then you should also check gstat to see if the > disk itself is slow. Hi, The disk in question is one of these: https://skinflint.co.uk/toshiba-mobile-hdd-mq01-series-1tb-mq01abd100m-a1820027.html It's a CMR disk. Power-on hrs = 39253 da0: 400.000MB/s transfers da0: 953869MB (1953525168 512 byte sectors) da0: quirks=0x2 Filesystem Size Used Avail Capacity Mounted on zroot/ROOT/default 863G 146G 717G 17% / there are no snapshots in "zfs list -t snapshot" rm -rf /var/cache/ccache/* has been running for 53 mins so far (it's still running) # gstat -dopC timestamp,name,q-depth,total_ops/s,read/s,read-KiB/s,ms/read,write/s,write-KiB/s,ms/write,delete/s,delete-KiB/s,ms/delete,other/s,ms/other,%busy 2022-10-16 16:01:24.761122175,da0,5,0,0,0,0.0,0,0,0.0,0,0,0.0,0,0.0,0.0 2022-10-16 16:01:25.762821956,da0,4,40,34,136,107.8,4,16,57.1,0,0,0.0,2,293.9,111.1 2022-10-16 16:01:26.763826447,da0,3,52,50,200,37.6,1,60,34.1,0,0,0.0,1,152.0,95.0 2022-10-16 16:01:27.767079004,da0,3,108,40,159,33.6,67,2173,10.3,0,0,0.0,1,12.2,85.1 2022-10-16 16:01:28.813081219,da0,1,57,17,69,62.8,38,467,31.0,0,0,0.0,2,156.9,108.7 2022-10-16 16:01:29.818093791,da0,1,56,56,223,17.2,0,0,0.0,0,0,0.0,0,0.0,95.9 2022-10-16 16:01:30.825841923,da0,1,57,57,230,16.9,0,0,0.0,0,0,0.0,0,0.0,95.3 2022-10-16 16:01:31.828940957,da0,1,58,58,231,16.7,0,0,0.0,0,0,0.0,0,0.0,96.7 2022-10-16 16:01:32.830822873,da0,3,150,29,116,33.9,120,8460,9.4,0,0,0.0,1,125.4,99.7 2022-10-16 16:01:33.877434519,da0,3,115,6,23,128.4,109,8607,17.1,0,0,0.0,0,0.0,93.4 ^C I'm not sure what, of the data above, what would be considered "slow", or why it should be like this now. There's lots of things in sysctl -a concerning zfs. Can you suggest anything to look out for? thanks,