From nobody Thu Feb 29 05:04:27 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlfK85VhZz5BvP6 for ; Thu, 29 Feb 2024 05:04:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlfK844Svz4TdV for ; Thu, 29 Feb 2024 05:04:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709183068; a=rsa-sha256; cv=none; b=ih0jl7szokMXbv+jhlpQSepBMWpJ6ISyThinQ8UKwcpsTn9rU56Z1aAYwFKmaa+Vv9mNv9 T4I7yfy8idVmWYSg5NDmmurAm17E3fsblf6gDm+5DUKDRWyITWB3GKF+mRQAyg6eR2Xp3U JpZ2yPA1kit10QFylfoEn9tnMFxddWqYTQUfUgcW+p3FBvNgmp5Tm6jUs8BXf+sgTi1lAg nbUDPF2V2vJw1GHp4wU2C92/+2AMkfYyWbdIwEGCPDmPjygoBHBQdzcTQ8KIQjQ3EjsMvS KqmNR/4jlAzPau300IecaIaTF1ncRE/Q2oNGUojGYX4wcF1ixzZI9XSiJDqdPg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709183068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=akaiFuPEU3jPOdsEkfeEJAc3vgD2VjytrclZ9JFawhU=; b=U/AcYZvqJuqkuwWD0rvFN2qiwqXvFgCD1Nnaz4pzJbzi9JJqurTeFkN3X857pMJNz6XN/l c89xVTNCx02pCt47G5IiI1aW7ayFIDlX/zzuvKiLGvxS+MKryV7+g0cqCBgjA40iZjig4I Rx01wXoc9AdeGm3oE0WbAT0i53uT+dQ/U7nv3C2lBPEnpSfZFutg13yNP7zoMVhJO2CaIV AaYBGZ2SVaZTVVFxUvcrHFexh01NDsP9G+BjRUdAznDY3lLhr2KgfLDsv5i3BgMAYCBFW5 KVMSKJHtpiIy6c9+9h6XlPMNTjdWPodGDiP8l0gxq2PJ+CEJWU8gaZAiDLLMNg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TlfK83WBBzSGs for ; Thu, 29 Feb 2024 05:04:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41T54SdN032826 for ; Thu, 29 Feb 2024 05:04:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41T54SA7032821 for fs@FreeBSD.org; Thu, 29 Feb 2024 05:04:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 29 Feb 2024 05:04:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #68 from Seigo Tanimura --- (In reply to Peter Much from comment #67) Thanks a lot for your test results and my apology for the delayed response. Your results look just as expected. > I'm a bit worried the thing is still reluctant to page out, but otherwise= this looks good. Some CPU usage by the ARC eviction, pruning and pagedaemon threads are still expected. My fix is meant to keep them form running out of control. On the FreeBSD ZFS implementation, there are two triggers on the ARC evicti= on and pruning: 1. The size reduction of ARC after adding new blocks to it. This is a part of the original ARC implementation by ZFS, intended to maint= ain the ARC size. This usually happens along with the ZFS activities. 2. The vm_lowmem kernel event. This is the case when either the free VM pages or the kernel memory (malloc= (9), uma(9), etc.) get low, and reclaims the kernel memory used by ZFS. You can hence see this behaviour even when ZFS is idle. There are some other kernel processes and threads that start by this trigger, including pagedaemon and nullfs(5) with my fix. Case 2 has to be handled carefully. On FreeBSD ZFS, the pagedaemon thread blocks until ZFS either achieves some eviction or gives up completely. The= ARC eviction and pruning must work more aggressively in such the case, while avoiding to starve the application threads. My fix addresses this issue in= the following ways: - Prune every single ZFS vnodes. (Work aggressively) - Pause 1 second before repeating above. (Avoid application starve) The results of this fix include the dynamic change of the CPU usage by the = ARC eviction and pruning threads. This may be seen more clearly by top(1) with= a fractional interval. (I have not tryed, though) --=20 You are receiving this mail because: You are the assignee for the bug.=