From nobody Fri Feb 23 19:25:52 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ThKjs43JTz5BMS8 for ; Fri, 23 Feb 2024 19:25:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ThKjs2zyNz53XR for ; Fri, 23 Feb 2024 19:25:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708716353; a=rsa-sha256; cv=none; b=kUUy1moYGgNEhFariTG0iGNdAeAELDsVeWrL+u4YgeYhUKQp08kXPGeVX70KLpAoMWHj38 RIQ8FUfuQZpvEgpNslZwH8SPhDhhYj59gFf4FWU3zuz0Mb93AxHZAKffVGrC+o8QpSghEZ 8EWu0xq5G+BbdXmtFiulfxmc+eI2mY4j0cBpGIFvtpZL+sx3WnhK3Y8E0NA74/hTHcnV09 hEt5XBvCotZDP1Igo9nEucGADQCFcTteYc7ZRzDbu50jTAC9x+AfFmRV5ZiM5n23s2PXRK NCdcT6Rq9Qc77nIdwR6+24GXkuNFTWul6DGIETDTwTu/Mfr2fDRvub0tLUkArA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708716353; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d+zIVOwMK7ROomW2qS5mt885i1PpL9CJQqNHf3AOcKs=; b=sK/DUwLH6E38gzXLEOy1rDgkVQCQ3ZcSByO7DNLm1Tjxd3UaBIsoMXoc3EvEhXveUoA/mH eaHf+X/oM0QAyZGIgp6gvcQ55F0aZiN/lf/Wbpe+WxSqvgAk5TX+Jk6rp1m37MtnVDVdaz /au766etxgYhTVnZrRri0MFIuHHWDNSMcm5+Z+U2prDbsonPh1Gp+czE5schOS094dJTJh RONqfgJYRdVJYurFRKaYtnf2SGXbQnu0CkBe2qIoQDCJJytpGtDbAcI8TXHGsZQxZ6jRsm AwGd4fBR1N1XTzowLEYyWKCbd7GTj6BHn7MNbN3wfHcx4XMSE326ShYNdhXejA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ThKjs22fVzTBC for ; Fri, 23 Feb 2024 19:25:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 41NJPrOd018106 for ; Fri, 23 Feb 2024 19:25:53 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 41NJPrgV018104 for fs@FreeBSD.org; Fri, 23 Feb 2024 19:25:53 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Fri, 23 Feb 2024 19:25:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pmc@citylink.dinoex.sub.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #67 from Peter Much --- So, now I read all the material here. Great work! I had upgraded my deploy engine from 13.2-RELEASE to 13.3-BETA, and found (among some spurious messages from git) that it can no longer build gcc12. There is apparently no problem with rust or llvm15, but trying to build gcc= 12 does reproducibly crash (10 core, 16081M ram). Apparently the crash happens when gcc fully powers up its LTO for the first time: last pid: 37369; load averages: 9.35, 9.93, 9.27 up 0+03:15:25 07:2= 1:42 417 threads: 14 running, 379 sleeping, 24 waiting CPU: 55.4% user, 0.0% nice, 35.6% system, 0.1% interrupt, 8.8% idle Mem: 7047M Active, 6121M Inact, 2392M Wired, 984M Buf, 60M Free ARC: 518M Total, 45M MFU, 451M MRU, 128K Anon, 3990K Header, 17M Other 467M Compressed, 997M Uncompressed, 2.14:1 Ratio Swap: 15G Total, 15G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -8 - 0B 2432K CPU4 4 3:14 99.79% kernel{a= rc_p 7 root -16 - 0B 48K CPU6 6 2:45 99.79% pagedaem= on{d 15 root 52 - 0B 16K CPU0 0 3:00 99.70% vnlru 37334 root 52 0 891M 789M pfault 1 0:37 89.24% lto1 37270 root 52 0 1017M 915M pfault 3 0:43 88.63% lto1 37324 root 52 0 831M 770M pfault 8 0:39 88.59% lto1 37338 root 52 0 843M 785M pfault 2 0:36 88.50% lto1 37333 root 52 0 889M 788M pfault 7 0:37 82.76% lto1 37269 root 52 0 1001M 882M pfault 5 0:42 82.09% lto1 37274 root 52 0 1004M 885M pfault 9 0:42 80.24% lto1 5 root 20 - 0B 1568K t->zth 9 0:02 1.02% zfskern{= arc_ 37360 root 20 0 14M 4940K CPU9 9 0:00 0.87% top This is the last output, at this point the system becomes unresponsive, and, when allowed neither to oom-kill nor panic, continues to consume 300% compu= te. Apparently these are the visible three apocalyptic riders (arc_prune, pagedaemon, vnlru) entertaining themselves. :/ Implementing the patch (i.e. five new git commits from the github repo) sol= ves the issue, and afterwards it looks like this: last pid: 11944; load averages: 7.13, 5.29, 5.77 up 0+03:48:45 16:1= 2:46 424 threads: 19 running, 381 sleeping, 24 waiting CPU: 67.9% user, 0.0% nice, 5.1% system, 0.0% interrupt, 27.0% idle Mem: 9308M Active, 2285M Inact, 20M Laundry, 3643M Wired, 865M Buf, 336M Fr= ee eRC: 1638M Total, 855M MFU, 575M MRU, 128K Anon, 11M Header, 198M Other 1305M Compressed, 2980M Uncompressed, 2.28:1 Ratio Swap: 15G Total, 15G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11579 root 103 0 1269M 1066M CPU6 6 4:09 100.00% lto1 11605 root 103 0 1263M 1052M CPU3 3 4:08 99.87% lto1 11589 root 103 0 1295M 1091M CPU8 8 4:09 99.87% lto1 11599 root 103 0 1259M 1027M CPU9 9 4:08 99.87% lto1 11588 root 103 0 1263M 1035M CPU7 7 4:09 99.87% lto1 11590 root 103 0 1287M 1058M CPU5 5 4:08 99.87% lto1 11598 root 103 0 1311M 1082M CPU1 1 4:08 99.74% lto1 0 root -8 - 0B 2448K - 6 0:03 6.83% kernel{a= rc_p 5 root -8 - 0B 1568K RUN 9 0:03 5.80% zfskern{= arc_ 7 root -16 - 0B 48K psleep 2 0:37 3.11% pagedaem= on{d I'm a bit worried the thing is still reluctant to page out, but otherwise t= his looks good. --=20 You are receiving this mail because: You are the assignee for the bug.=