From nobody Sat Jan 20 07:41:09 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TH7hQ2Cv7z58Sgx for ; Sat, 20 Jan 2024 07:41:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TH7hQ0txsz4C9k for ; Sat, 20 Jan 2024 07:41:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705736470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P8OMGY/i5RCBzjXb4GsMZ2rUxU1SQPGED9KI02djgUg=; b=UivDIlZdtMjTplTzwKKODeXY1LOWIujJ8O4vZ/QXBfefL0iVumJ+hEOsi0l6PGcvrJqDOF 1d5vq8VCW+Mikj6mcrq2mrATSca9BnbO6zZFGOWZEGqY7b5JP9bzl20eZasBcNONG/ITvR xDWRQs9bHskyuVejllMRdJgz4tqUMvIBjhwOjKPctCnTEHFnxW7JFXoJdwqtgIoHFIbBrL NKDt3vfxbJQ7J2lsPfrYvEeN+J9hcidHRL4JNPHg06kqcfV5/lB8LOfDsEM0PsO4DzaslX 92u23UpAzvxiAu+L2POXwFhtY3IqKqTHHi66rYQSULtR+YHaN6jXwUMbks7IIg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705736470; a=rsa-sha256; cv=none; b=sae74LFGYDvfEL/og791vQE1MZ2L76F94vByR2T9lDGj4H3psukz1DfDwqx+syLf+X252o Nvfgi0B3HXyOMo//ebRU7wBY5q51k8NXVi+JrWxcwE0b9/4gG0L4+2PgIUZ+KPApGbCvCr FGLImxK7CKmqW10gbQPgbZqjat1bwXLpI7Sya0wDC25ePRxTrNFkoTYE7UzEDEJk2KBBvY 7bfOWBRlq5XqDs9rFsQjbCGQcH4spFXlbrzuUnVWtQHBZY/ts/9PyPaFlcXxbfpbdYnKqL sxULOytfWLQww/3fcOR2p1xz5KkooqSH7XByRAR4GkP4icuhNHXcnnjx778vVQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TH7hQ00k7zR5t for ; Sat, 20 Jan 2024 07:41:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 40K7f98K065343 for ; Sat, 20 Jan 2024 07:41:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 40K7f9JR065342 for fs@FreeBSD.org; Sat, 20 Jan 2024 07:41:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Sat, 20 Jan 2024 07:41:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: thmu7@freenet.de X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #34 from Thomas Mueller --- (In reply to Seigo Tanimura from comment #33) On Sat, 20 Jan 2024 01:43:28 +0000, bugzilla-noreply@freebsd.org wrote: > Do you see any other threads using the CPU as much as kernel{arc_prune}? = eg >=20 > - vnlru > - Any threads that access files somehow while running poudriere-bulk(8) (= eg > cc1) Yes, I've observed vnlru CPU usage 30-40% for longer streaks occasionally when arc_prune was at 90-100%. With 12-STABLE it was possible to have poudriere running at idle priority on two of the four CPUs and use the system for everyday work in parallel (X11 UI, MUA, Firefox, or even Virtualbox). With 13-STABLE, the system bogs down, video playback drops frames and/or audio, etc. > If so, what you have seen is the same as mine. Kernel{arc_prune} and the > threads above contend for the vnode list lock. Each of them spins in the > kernel until it acquires the lock, which can be found by top(1) if you ha= ve any > idle CPUs. You may have to reduce the builders to let top(1) work. Exactly. What's also new in 13-STABLE is that sometimes when the issue occurs the system runs into memory pressure, and pagedaemon can be observed with remarkable CPU load and processes with high memory usage get killed (firefox, virtualbox, for example). That might perhaps be caused by some changes in poudriere default configuration, so I can't quite tell whether that would also not have appeared on 12. What also wasn't observed in 12-STABLE, occasional build errors with=20 "bad file descriptor errors" which then cannot be reproduced after restarting the build. Example: [stable13amd64-default-job-02] | `-- Extracting python39-3.9.18: .......= .. pkg-static: Fail to chmod /wsgiref/__pycache__/__init__.cpython-39.opt-1.pyc:Bad file descriptor [stable13amd64-default-job-02] | `-- Extracting python39-3.9.18... done Failed to install the following 1 package(s): /packages/All/meson-1.3.1.pkg *** Error code 1 > I was not aware at the time of the last massive poudriere-bulk(8) on > 13.2-RELEASE, but it is now likely that the same issue occured on it as w= ell. >=20 > The comparision of my poudriere-bulk(8) results, both on the same host ex= cept > for the OS versions: >=20 > | 13.2-RELEASE | 14.0-RELEASE > -------------------------+--------------+------------- > Build Date | 13 Apr 2023 | 19 Jan 2024 > ZFS Fix | No | Yes > # of Packages | 1147 | 2128 > # of Successful Packages | 1136 | 2127 > Elapsed Time | 18:44:33 | 06:54:28 > Packages / Hour | 61 | 309 Looks familiar. > > Questions: > > Would migrating to ZFS on root mitigate the issues?=20=20 >=20 > I would say no; that would give even move pressure to ARC. Thanks. > > Is 13-STABLE in focus for this PR?=20=20 >=20 > Not for now, but it should be. In addition, FreeBSD-EN-23:18.openzfs sho= uld > include 13-STABLE as well. >=20 > I have one baremetal 13.2-RELEASE host with ZFS, but it does not suffer f= rom > the issue as of now. This host serves the volumes to the bhyve(8) VMs ma= inly, > so it does not use vnodes heavily. Thanks for analysing this! --=20 You are receiving this mail because: You are the assignee for the bug.=