From nobody Fri Oct 06 00:54:00 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S1qgd66Rwz4wQhr for ; Fri, 6 Oct 2023 00:54:05 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S1qgd103Wz3dDK; Fri, 6 Oct 2023 00:54:05 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Iz6l0tIY; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qk1-x72b.google.com with SMTP id af79cd13be357-7740cedd4baso108869785a.2; Thu, 05 Oct 2023 17:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696553644; x=1697158444; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=eIzg/AdZ/rRVdAxYrrIzUeizHYDVr2N6bfG3Hy6jcWg=; b=Iz6l0tIYuZNQt/5Mv6VvjqxTOwRTDnHclSQfvu9ULZj6y2MiOpfZpRbdJFjZ4bsOlH Db1pkyw0b7GLFwfGrSLSzy2lg40cp4ROkGLSPv3MXoAQWvCk0kB18iLBUI4tAhp6B9KR WKEHEzvIp3Nh4m/zmMc0UYYkTLlpubDEk1v0acpWB13Q3olZYesWS1GsPw7VqDltruYW ziCax7RPeppamm0FmhCrIlq74ZawviwDholaZLVxhOVfmCQx7kjFi/xaKMDc6iQjAfYt wmmQAfO5jQbVbeW3J0BcClt3O50g80CADk0YeXHJ4gH2T+SMXtXq1DxW7BlpajTCc1EB WLcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696553644; x=1697158444; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eIzg/AdZ/rRVdAxYrrIzUeizHYDVr2N6bfG3Hy6jcWg=; b=xBrBBuYoCL72HmDI4h1fwNXnMKTt4jdtZ5IY7iTohHrmcpGDSbunn7r2R8cB+/mRJ+ MZg492lSeP89yZWGUNRRFaoVqEO7LnasPnoiV/VTgg4d8ADKUK8IO0HPyBUW/TVpu++C cU21NEYimYEMlnPP/13bE2GOK6Y8e2kwOYKFkIAHkL+3YQrKI5Tfie5jo0u0LMY5m/Up bLghRhJyaRMrcTBL4EE8lVjpjpajjk+jC1x7mJKzu0wjglXC+6NUr97no62p/swTU2YM y55EWjhZAwm1ONwU8meIIUmsAaAkWkzBnw77QFzwb6T+VEKSXSw6Ei03tzVAil7w7WQl dGQA== X-Gm-Message-State: AOJu0YziJ2AeaFM9+I3Bh/S6YJvHdtuPLnFwnETOXW5y7EgiWKKFWXl3 YACl86XjNc+GaZoEgsrq+48+4hEnWUo= X-Google-Smtp-Source: AGHT+IGNDs+iVOI6z5XcXrOLVyIf1uy7bcR5K41aXjr6au9C1gMITuR5JuuceS8oIR4B8BZza1A1lg== X-Received: by 2002:a05:620a:e9a:b0:774:1494:bb0f with SMTP id w26-20020a05620a0e9a00b007741494bb0fmr7092744qkm.60.1696553643704; Thu, 05 Oct 2023 17:54:03 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id x13-20020a05620a14ad00b007756736aed7sm894190qkj.106.2023.10.05.17.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 17:54:03 -0700 (PDT) Date: Thu, 5 Oct 2023 20:54:00 -0400 From: Mark Johnston To: freebsd-fs@freebsd.org Cc: mav@freebsd.org Subject: arc_prune thread consuming 100% CPU Message-ID: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.65 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.951]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4S1qgd103Wz3dDK Hi, Sometime in the past two months I started seeing a problem with ZFS on the latest FreeBSD main on my desktop. The symptom is that the arc_prune thread, which attempts to shrink the vnode cache, consumes 100% CPU even when the system is idle. I was confused at first, since dtrace shows that 1) arc_prune_task() is getting called several dozen times per second, but 2) arc_prune_async() is not getting called. But now I see that the taskq zone has ~250000 allocations, and the number is slowly shrinking. So, I believe arc_evict() was called many times in quick succession at some point, and it queued up a massive amount of work that is slowly draining. Could there be some regression in the arc_evict() changes that were made earlier this year in commit a8d83e2a24de6? If so, I'm not sure why I'm only seeing it now, it looks like that commit was merged to FreeBSD in April.