From nobody Sat Jan 20 01:43:28 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TGzlk1dj8z57t51 for ; Sat, 20 Jan 2024 01:43:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TGzlk0cktz4bM4 for ; Sat, 20 Jan 2024 01:43:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705715010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OPciEzZRYJGOtfQUHMjR1Ntg26UqibdSnDbGal0zALA=; b=ZgHBxK0JEEKJipoP0BSJYwHObPer+Mg//52Z8ax/93fmJf0x6BEPLBgssAJcS4mVjHCaem bvX63c7ADL0lYIBZpubxZN5iOhu381ayrw8J+98d1GN/I/Iko5zkllvStYZCYtvHaSF7Lg LDLnwT/CBfMi5yMR8wlFb1tMjafLl+eMfQLO12ohN8ILzxbkrWH6zjMsn9gY9pel9ZnthE 5sRaalzOSWaDMYD18XJN/2gZ77DAoFL6Ij7FD69zC6nxlGpGPDmxu5FazOlScvWI7iLv/s hMNBff5++ol0u54vRl+R9oJu78GBmc1cQUO5gYHQgCTe1onLctV97E2/fKZVHw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705715010; a=rsa-sha256; cv=none; b=a1YvyoyotwGGez8RauyTwWiuJ7U9SXJVj8fgTZPvdO0yWwAG2IFYDp0kSYPABGyqmp+fkO XjkS9vblRvS5EhtBY3yPKwlsKTUsN7A+TM5bUDpSyGaGzz73fELkQZlDPUgOXANkhNx3b+ x6XmGHx98wx2k937pPIc/sNC/4AXJfUgbJ0yM5DaQl1AHraFs+4j5W6JKCCZMSBGXQJULl hWHdT8SP5jzKlsTB+pt07s6ordR8q3MZsAsWMXI0osSPhBQ1jU1IC6W7O3dcJO0svky6Lw IRNg+xSRv/HJ944aTMp7tBX/OfY8IPIKJpdC/xYnEoPtEVEh3jDMZ69BdLNdDQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TGzlj6pJCzGCm for ; Sat, 20 Jan 2024 01:43:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40K1hTKq066567 for ; Sat, 20 Jan 2024 01:43:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40K1hT7Z066566 for fs@FreeBSD.org; Sat, 20 Jan 2024 01:43:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Sat, 20 Jan 2024 01:43:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #33 from Seigo Tanimura --- (In reply to Thomas Mueller from comment #32) > After upgrading from 12-STABLE to 13-STABLE (due to ports changes > triggered by 12 being EOLed), I'm now observing high CPU usage of > kernel{arc_prune} on 13-STABLE too. Do you see any other threads using the CPU as much as kernel{arc_prune}? eg - vnlru - Any threads that access files somehow while running poudriere-bulk(8) (eg cc1) If so, what you have seen is the same as mine. Kernel{arc_prune} and the threads above contend for the vnode list lock. Each of them spins in the kernel until it acquires the lock, which can be found by top(1) if you have= any idle CPUs. You may have to reduce the builders to let top(1) work. I was not aware at the time of the last massive poudriere-bulk(8) on 13.2-RELEASE, but it is now likely that the same issue occured on it as wel= l. The comparision of my poudriere-bulk(8) results, both on the same host exce= pt for the OS versions: | 13.2-RELEASE | 14.0-RELEASE -------------------------+--------------+------------- Build Date | 13 Apr 2023 | 19 Jan 2024 ZFS Fix | No | Yes # of Packages | 1147 | 2128 # of Successful Packages | 1136 | 2127 Elapsed Time | 18:44:33 | 06:54:28 Packages / Hour | 61 | 309 > Questions: > Would migrating to ZFS on root mitigate the issues? I would say no; that would give even move pressure to ARC. > Is 13-STABLE in focus for this PR? Not for now, but it should be. In addition, FreeBSD-EN-23:18.openzfs should include 13-STABLE as well. I have one baremetal 13.2-RELEASE host with ZFS, but it does not suffer from the issue as of now. This host serves the volumes to the bhyve(8) VMs main= ly, so it does not use vnodes heavily. --=20 You are receiving this mail because: You are the assignee for the bug.=