From nobody Sun Oct 16 18:24:05 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr7n965lmz4fmv0 for ; Sun, 16 Oct 2022 18:24:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr7n84VVTz41Pj for ; Sun, 16 Oct 2022 18:24:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1665944650; bh=31zG/RLKBCHB2GktVmBzzRsQrBkLCNUXVUB/9WRoIwg=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=itUMEUNwr3L3iJG7ik3hT9jTiy0GRFYYrZRN+TR9xD1VrCATWJPvMwP5N/o8DcClSB2Y0Wmlwpuo2qZ2KL3G03fRKRLRElZpMv+ch4Qh6Rm2Z6zRvW0PoCnKqkcuonoOF57UTBPhQM58qle7GuzQT8Z/bkd6d89ia+PjcOQvR1wV4Ymuu87HeUGI9qfUCozzogYrg1lQClihsWgfcnfLy7u+2WGc1fPW3XNkGbE6uErXKqyJWwDFcxUVGysy+XMSTC8J/5648sIKhsMZITpiA9VKvUucUZPfaCoF2sQDzdrTgwfhrxrpUF9cafqr3eByXC1/MvfCns+m9ERABNuacA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1665944650; bh=QqwyPobxrnsdzJhwn/VnrJ2i0XDqnhlH5BlPXlNRoZM=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=pVkrnmliQFw0MiFyOfeq+H2BmNXrtVwrbLetdG61xxRv633l2xM8p/h33mKqXK8/LTV4FysoVYDYJoFRTnluK/BQugSBeTgbAA7wm573/HIVI7NWk5RtDUUnYMQrIcISbYblsaiE54vQl2wdQNntFuwsBaCC0SxvkPrNNy0bm2WcQ00TpxALAhXd/ZbnUZDSnbAmd6NMWt/jeE0pF+p/TYyF2eFzf9n7IQLPfNpylc7fGnVMDV3ffTKIhk4+k0na04gWR9v6aR4XI9m6DIUV80hn4pLmyIqAdOBnwCY6Kn5Zxl05C3DeU4Z2JWND0xWWAoZWeNQBJKE1Hl7DPa+Kjw== X-YMail-OSG: c7wvOIEVM1l3oSfYkZqSUmJEUKwO0fbY4SVQ.OtPSkngvI5lUgg2pctIE.fco1H pOe1DXIv6gC5ZIT0BW7HrSKoywyhp_mMkXzSKpdKFELpQ3lG8F43uw6kxn5ykUtkWmBQLoYDXXqz mKwfQ9vETNGI3oGyiNfPIUf1VsZ1HTtTnrpC0BE5saHXlU7Et7rpd9QkSnRRkiAna1IQwDUy9G8e 97jMa0cgwS6ZKpIfP4a18.Dm5zGmLGdATFvDbj9w0gdYt7tP7bNj0e7C.Vpnfoo1EVttBO49mlau LxWMt7chuDFTd2BTe2t8Oy7OKRWCWhLQC_pA2PkkTgSWN62uQVMzYeFRTk3Znt0qMVke6yR1iqk6 OdJaKgbOLSWjU.n2AYe5kcPQuLkd36rKAw70BNNRy3D3NbYCxjBe3bhsvacz8FOj9Df4kEpPk9y0 t9daQpnLS8_TaiSEGdxJvJntzMLDddu.mo6fuVUfnX6mmBZNL.cWA9Z_Qezb6ymlixR2Ple7BmIs h0YiW5bs9qNGPMn9vOIJOygklMX.LRJnuaCIbd4h0jexOvweHbciooA2gMBn0UaeVUFg4xtlIPM5 tLAVsLWVx0XK4W8GRxhOWSmpnYPk02yMhdMaPXj5QFUDMOtMJBdIEVCALvfzSnZjnQXQmG53M45z eop8zJHykrOVpLK58hRu_AQiy7CIIN49S7JSNL1n9MlZ_dTzRIjC72bu61tsbUZL8uteMlLxc7Yx nL7zvpp7pOxctjSNM_GkYJX2.3Tlj4ykQXUpbjgXQ0s1tcaW8d0sxePOLtDSQXJEIMUgsUWTPv7R qcLIKhNzWkS3Ok5rIyZduNDxdUO3jekYXiSYzNX8dfL8B_CXSQ.eW3cULIAp9ul4_uVAi6z5V.YS IDHU.ZzgIJmmeb8SoZ8yG17sZINoZEgFrjX7zf76HrQTc6l4naWMufaWBSrLFOOc7hbOadQlttKp QYFRvwcCCoZajXp.vyDg_WsFVJntF1DmrlYayGpj8uPou3FhqKdIa7YgkYGewS4URl_NwGM4edUh LqykohcfzaZnmy25J.zNvNfAX4i75RnO2mFplM9MXREPsShIw8Wh7z2qGJHY3sWpyUqfG0nW3WXD wuUuncAcoAJ7Rf4xJmcetcId5GAaOasM6O1QJwOiSoT_ZPzzWcPOyjE7nyDivytm1WTuCx90HkBD Bt_B944EunvGIiW.oqaPlILeoBXL2plxrS7H8BGqmIlzSV1n6eAUO122wMbtoYC9TYOYPi9qTLoK DCdrSUQakFMc2GxY2YczzTk0x9nf71cNLBLQ8ujorR69KbJCvJ8PQXgL__aRQfrH4c031QFRgB2c 5X10fMAeAFhqZiTip5VYO0H1j5_bTF_L_FlJtWbMLtgcDcazDa6kIcuB7NsvxOoUHnEeH_JvxtdH bt8S_csdJOClrFWMBknCduuR.JP0X3jYf7bkwU8GOC.4RipUgb4mRYouwU3RJAXBAewv3Z6oF3Ag GyM81tCfH.MZISNIad23yN4.A5ymHKsQz8u140l2JmQ7AP71LyfNzyCgoQpts8i7QpAD4cQujJkj yxL4a054klkDlqZA41lsXQEc.PEKmHRvS16KkDesTK4smnWrbI3iQrJEIG0phCUDlxx5ThKFoUEl 7TEvqVPADX.fuMw.5Uzee_HLbHHepv3_2cYse0CqRcPXCeBW9HGv.XfOs92wCHw3qP.ltfgeRR4F .f.Ng1hxIil52SkaL.Opq_n3uGtcDA1uDyInS5SuDK_Yx1l9FI_M6lhAdZ7.qPO3I.FQe4xDPHOB 7.J8wkPml_Z8nUC6rSnlBYGwpL96m5aWgREvkDz3VjppRh6Np.8nYuRK61DXe8Ga2p3IR.oKRc0u gVN1FLit7WNrqckjw2yRYV_sJr8VZn5kH.MOb4W4sPnvxYvd0m1w2Bq5HuRVazkdvSANrOgiGhQQ GHGIivnpxPYFVxEDg5QvB6ghKGHrKBtVOEueoyAIpRcAkY0PO70EyMuvCNHISv_9ghnUo8Lx48Ta mNr2548OR7FMZfvuB5LaooXH9UqrqHP5tw7YHZ4xhzyc0g1Q0aW_VivHTlW5rvSZ7CSZ_twEMlyw fO94ecYbjhqueglktaz5WelHsKJXjVPdtk1gIVsfydU7lTh9g_NmjbfygSoFP7eq5f4mAZ1eHZzd B0r1uLqppmWhQkzhDe7CeEXU7yPxPaspdSEKuxVAexzKBo28f0zKIAgX8haVF6E6LBZ20Uc27Xdl dQKvurjJ7T_1OBrmXWsylg3X6HH_.re67o4PThMKePNTXG59M7JCrHK.su7BnEmgd2s0tJQg- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Sun, 16 Oct 2022 18:24:10 +0000 Received: by hermes--production-ne1-5db649d989-tfbvd (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 94c3801d89b9741313b68e3da2c86841; Sun, 16 Oct 2022 18:24:07 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: zfs with operations like rm -rf takes a very long time recently Message-Id: <738D20C3-2F31-4869-B318-F1B30CDE3764@yahoo.com> Date: Sun, 16 Oct 2022 11:24:05 -0700 To: void , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3696.120.41.1.1) References: <738D20C3-2F31-4869-B318-F1B30CDE3764.ref@yahoo.com> X-Rspamd-Queue-Id: 4Mr7n84VVTz41Pj X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=itUMEUNw; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N void wrote on Date: Sun, 16 Oct 2022 16:48:39 UTC : > On Sun, 16 Oct 2022, at 15:42, Mark Millard wrote: > > The book is not explicit about RAM subsystem performance > > tradeoffs for ZFS. One property of the RPi4B's is that they > > have very small RAM caches and one core can saturate the > > memory subsystem if the RAM caches are being fairly > > ineffective overall. In such contexts, multi-core need not > > cut the time things take. (But I've no clue how likely such > > conditions would be for your context.) A cache-busting > > access pattern over much more than 1 MiByte memory range > > drops the RPi4B performance greatly compared to such an > > access pattern fitting in a 1 MiByte or smaller range --no > > matter if it is 1 core or more cores that is/are trying to > > be active. > > That's interesting; i cant understand the system has been in > use for 9 months or so without this performance penalty. > The OS has been updated on the following timeline > > main-n258595-226e41467ee1 on 2022-10-13 > main-n258157-f50274674eb on 2022-09-23 > main-n257818-6f7bc8e7a3d on 2022-09-05 > main-n257229-e9a2e4d1d28 on 2022-08-10 > main-n255150-70910e4b55c on 2022-05-04 > > cleaning out /usr/obj and /var/cache/ccache/* is something i'll do > periodically if what is required is a completely clean from-scratch > build. On at least two of those occasions I'd be doing what I'm trying now. > but only now this issue has arisen. Of course, it may very well be > the hardware. But all the things i'd use to monitor it say it's all fine. One thing we do not have is a set of before-the-problem data to compare against. It is hard to tell specifically what time frames have changed. > would output of zfs-stats -a be of use? Earlier you wrote: "Right now it's rm -rf-ing /var/cache/ccache/* which is 5GB max". For this context, the number of files/directories is likely more relevant than the total space the files/directories take. My experience with spinning rust tradeoff management goes back 15 or more years at this point, beyond certain backup storage use. Even backup usage of spinning rust is not in the recent past at this point. So my help is minimal for such. I'd guess that each/any of the following could produce interesting background information during the problem: # zpool iostat -w # zpool iostat -l # zpool iostat -r # zpool iostat -q ( See: man zpool-iostat ) === Mark Millard marklmi at yahoo.com