From nobody Thu Feb 08 03:32:05 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVjGH3rWxz58l6s for ; Thu, 8 Feb 2024 03:32:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVjGH0l7Yz4r9v for ; Thu, 8 Feb 2024 03:32:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1707363127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qh3cmnFUrtr4ow/oVpcvkCqkpgaePdAYs8Aa3D14CQw=; b=Bd9/8J9CXrtJ46DeCkXIqCG26Vt4txJfs6XaQg8uFGdhUQYoeCXNlMDcJRoYqw7zwpOVTd ucV+G+6UcF9MDjEO4YsdrWVEphPGQRSbiiTIdcMSp9YvSHvHQX612FgwDbxe39AmdvWrBZ qbiqB5hu1YSL0FsddALBFZ1twIBznvjg+qM/CCR3taIscRu9DB8RD4ZHguFVmi8BsOxTzU 4i8iY+2XXzxqjEklRO7Xt4Dt0jQfeSed7QpYy4LO83A4hJXkJWs61sPsptKbOeCj0GEYRt DHr26uVcgfxYMPY+vYFzCDh4Nfksqp5Lp4hF3wB+ywpbxkbAtI6+7cJ5qSCmSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1707363127; a=rsa-sha256; cv=none; b=plmfCDKp9Bu8QkV+yjvpe1l2k7/ud/Je5Y19htKhY543PqWknB+Q5hKWutm8VzTAnAOJw5 Avs5dNZLk/lANV0YaycPctsH9YFRDllYlXvOqSISSlCkzAxKeB7FyfPjq/brOcnMAdW33U cPUmNbPzx62hstX0h1rXK795Ct3O56D/zBc5/rboPwuDq5779HIXAM8JQh2VEiNuS5te/8 Tf+QigwPeCNke2G1kuUtQ9Fefs4oY5fcmQNFT/YniaRv5mOJfaUoilihgFHLADn5GdvpAq 0tnwhnaNqgxwEvolLq3QhX9fu1Y9Z1L9u5BFfT1enEvQGXLDK3BNv52KZQOa/g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TVjGG6wW7zZLY for ; Thu, 8 Feb 2024 03:32:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4183W6g6007743 for ; Thu, 8 Feb 2024 03:32:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4183W6xW007742 for fs@FreeBSD.org; Thu, 8 Feb 2024 03:32:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Thu, 08 Feb 2024 03:32:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #44 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #42) * Fix Status - Backport to releng/13.3. Done locally. - Stall upon low memory When the vm_lowmem kernel event happens in the situation where the ARC prun= ing cannot evict the sufficient ZFS vnodes, a pagedaemon thread may wait for the ARC eviction indefinitely. This causes the partial system stall. Accelerate the ARC pruning in such the case. The fix has been tested locally. - Nullfs(5) node recycling This is the fix targeting at poudriere-bulk(8). Recycle the nullfs(5) vnodes not in use in the same way as the znodes, so t= hat the lower ZFS vnodes can be recycled as well. The implementation is partly shared with the accounting of the ZFS in-use znodes. This has made a drastic improvement on the ZFS behaviour, including: - The ARC dnode size has reduced greatly; it no longer grows monotonically during poudriere-bulk(8). - The ARC metadata and data now always have some evictable sizes. At lea= st, they no longer fall to zero. - There are always some number of the prunable ZFS vnodes. I believe this is how ZFS is supposed to work. The fix has been tested locally. - In-use counter overshoot and undershoot An overshoot on the nullfs(5) in-use node counter (introduced for the nullf= s(5) node recycling) has been found. This may cause a wraparound on the vnlru_free_vfsops() argument and hence make an out-of-control behaviour. The fix has been applied to nullfs(5) and ZFS. The local test is in progress. I will publish the updated git repo once the local test above completes. Hope there are no more blockers... --=20 You are receiving this mail because: You are the assignee for the bug.=