From nobody Fri Nov 01 10:42:37 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xfy9q1km9z5cD0Z for ; Fri, 01 Nov 2024 10:42:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xfy9q14lZz44Hl for ; Fri, 1 Nov 2024 10:42:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730457759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=hNE5QQJIzj95KAlvq5SUQwjcCn0SR9tyNQ35IiAub5c=; b=dw55AmbbV3HeSyPTg4/HuN6P6a4lrLLtv+CJfzmnSqg0WBERhAbmSZIFy+SRzxgAvtvc7U 5uW1waFGAyIiqhDyjSl09BkXN0RAIPGbEVD73T4Yiw9KiiAwnNPDbT/b2luajt+d0dwiYT enb8gTPiY2mIuVf/Tgw5G8caz5xo01mcIa2Y8Xo2x3auZKTEhcSn4NnC8bdz8Pcnb5fDEp +FnlYnTzgcqV7YiXnhQCggYf/xwGCUYfKKrS7nHiknAho5V4SgxaA5SCNC5ZnIPToirFM5 rfp2gh135diLGbyDgfzLThSqz4OhQ4raPiigQT5NDSJ6UZq2Qrkt34Cn/+3ENA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1730457759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=hNE5QQJIzj95KAlvq5SUQwjcCn0SR9tyNQ35IiAub5c=; b=bLl1vr/zuVWBXrPsGDPow7cssNwF41or0K6mBhtnogQAj3xIb5ALGyMZHJLMX9ZcHidu1x osG0AHEyesyA9fRa/xYr2WIIml9+WuP3++xwT8WydUAYbbG8dxhcLVnSR5oY0xg1iz6x1a 2zRWNQGrz9RmCGpF7aahJ/pGuWRTFg2Z8utkwT11YLgZVABGrGNAwTlHjUn3UtQF3EIIfB WoZfg2NRhSKWLaSCqYiFDEsMfiTPI+I2FXSCn5Dykb71EJDjQl5RCwRvkgsJR+bka6Xbc+ RDL39gV4ArtxWlH3RfAbc7ztj03m0nwQ84MbJIVg+CSY7ms5PAyLQrqjX88BlA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1730457759; a=rsa-sha256; cv=none; b=sU+JkbZfhcWgoH1AgwDlR8VyHRZIiB6GbauaRw4fDO0U9e7ZUQaGE+kvPsi9CaCQjYql2v EeXiDfTrCyAvY5KUMhl8UAJV7C+A3NRTDB8vE58k6noNLl75sszKp70cIS/rBgOYc2RLKE yx6BXTjbbrwRAEU7YzhEKBSnOfhogea437uRftK2sdmWylRyXfQaRgHHfuWagGlQhTAI+m 0OVVpIzzuDFa40Zs7AkgqIvr0fTli2gyvb9hmqLz+WUQTrx0+xx7ZG6vEAIdLH+RvUBA+9 6XKtSQYfGIpp6Jj5DWWfUzwYofzNJ8DqPoinADwsjdHjauod6WeAXtgY9ULWtw== Received: from [192.168.0.88] (unknown [93.188.39.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Xfy9p68Xgz1VmD for ; Fri, 1 Nov 2024 10:42:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <7b6475ca-ef66-474e-9084-2bc61301c31b@FreeBSD.org> Date: Fri, 1 Nov 2024 12:42:37 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: vn_alloc_cyclecount is always zero? From: Andriy Gapon To: freebsd-fs References: <1181f483-7e88-4333-9b22-c76146ec82b2@FreeBSD.org> Content-Language: en-US Autocrypt: addr=avg@FreeBSD.org; keydata= xsDNBGcKrHEBDADRvwQOK0b/yo4ys5cs6bOQMhEh4xtfbaZ/CU00cpPgUip3sOZCdrtMWlRC g25z97prxE9pKueZi+HXDhIPpa9xl14ghqF4oYScuJ1i18HyiOH2y5Q3Vv/TtFiSzicd3EAu QgS3jVidpgDSPDdj2Yz3UxYpZ+PuFl6nOnvCvqOFcjUlzKCyPaiN2b86l1Nscmhnc+zQ/faB erUOEFEDQbWMA5YfXi8HrbeR16hfRfGt7E0aMDlIj9FIPIq71UWMN9CimPgs4+rbNr1MAlLa z4GxSDhVYZEY5rqtCzr+PLXboRQWnaUwXl0/biw9enf17NHdYv1SNAFTX2eC4dZ3qBVI74dS PgNprm+PMfz+6Hhs/dAv+Nan5nVhg3EFIjYTiy0MnjMSq8uI0v0ykpAGAcJJ5xl6d23aLxgN 6f0z6pJRCO0hGPgU7UzvFD0MxJxmbzqdT1R51KDan1oD41b+tjl2LMBuCDCoB0U44Pu0zLdp xMfFTxCXtwIYKIUxwd28jwMAEQEAAc0eQW5kcml5IEdhcG9uIDxhdmdARnJlZUJTRC5vcmc+ wsENBBMBCAA3FiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHEFCQeEzgACGwMECwkIBwUV CAkKCwUWAgMBAAAKCRDMM63k0uPru5tSDACFK15LLbq89RSQ6QMnjiIm1t/wYJyumb519MHu Dhzxx1lbr8oghf0RHtF6kYRLQPaW2VdToi74pRobd3CN4bhZKDLSL6WfTn17RfavDjL6Njwp KBo30CkOeYKWq1mDmo0xEoQj8cc7ybEZnus+YScZOpj8Ti4EFwhRt6SHer7YDb161IHKL8m4 MsCxpFSGEjbKj8Iul3Ri/fTOO8w14ivcuEEQIvJt4/+4YV5Az8G23wKzL/3aJ7SOT3oYGmR9 atBTmVO3DlODjM+rZLegd8SfLSPTcBTHspWE5duemIzZbEX3BP77r3Qx4Fo5Tkit3bG1XVar yPQato+sFGFEGifdE9USBQoAoOaaeZevwAWjDU0TIuCT0CUe0sKtQuNP4LRq0n9EEHOXBu9a CfdMhFUSkAZnuE7miSVwgPvoVNJ1stA37EXLN/sVsWik7wslTQ5vF81VpdGFiwoQPOe2XEKh ogcwGSnXbwv1gD4x+Gz/7Y+kFyr1NY+4/nSaeXVcS2fOwM0EZwqscgEMAMQTe6ypAmQe/TFO HqKD2hfFKdksTptKi6uEh8xIwct8G/0FBldDWXo9eu8CGr/ZrDg0/bAwJxbaLRQCMH19Gq2Y hLvZ1QK5GQJVzZKcqfxbF2LiDUTs6WkdOBIhGpdDy7p1xFrvqCGCtNFYHuGYm067EozibBSF BWAPstKu2FQuVHZNMOfs7p3OIz3Yfqu9woXDeg3/8G2qVQJINe+8EwXKlhgh4CyDbq7nAZoA kIu1SE9z9u3WI5mcNy/0dFmVUsFxBqRC3ewbvzie8tKyZ9yFOlaZPT0Y4nRBXQTI3mLZ8zQ8 mtrWK5OOmrJ02kdeO9RBXe+OMaUUWMf92ZIoBFb4HP6N+B+4N1y1OwULousfl7JRoYxA4MRL ls7E2sSoJvrEBTJB3Pc34xu8rsJ1A5V3NgN6djX8yEZYpTRkcmrBeWy/ofDqZPVqneAx0LRm eldDS9msXDW4KXODyPZ+9unvmHAcoH0xaBYaSH44CDZDQDg4LNcmbOvuu1TEXBJhjQARAQAB wsD8BBgBCAAmFiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHMFCQeEzgACGwwACgkQzDOt 5NLj67sUCAv5AXqgWnYN9EblapMbZjkiqL8pZQ0GNqh+Pg9FwbyULxjtRTO6rD4D0IxizByb ef+neeUNyYlagt5nfKMysEr0SU/gHKCi8vyTF/63ukMrGUNGmJJxrndl5ZYKC6j6eX7twrZF L1Uvlmn6FnQ22red5kHO93fDjG4zaDIZvHfwj7kzjZ4tpC7Byinf88s14mdZeScc0PnU2hj4 UGYju/wg2FF4YxaZYhcmdTiRYY0Wx85XSMZv19pnn78sadEuRvfRd4JTmw++j1xGXeqQGWzz /CTG5/Ex9GAkQ02hZbmi236byDXoet4G8TEyOph9QFVkV9bNd0jQZaFZPGEj4PSPUYGAF7s5 xJaNGgctC3aZ7WjEv1FBoo44XCU4xcjJ1wZQUrHxRhx6TW0Jtcl0U9qfKFW30TSPo6RyiXuj X4ltWKAtjoXB8nUmEJckaz7IRu2b4pXDeazZuz5JBygUs10yJjDxh2vFQZo0KaBAPx9MZlPn gpPTjT15L8xGftEjQXF6 In-Reply-To: <1181f483-7e88-4333-9b22-c76146ec82b2@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 01/11/2024 12:24, Andriy Gapon wrote: > > It seems that after commit https://cgit.freebsd.org/src/commit/?id=8733bc277 > vn_alloc_cyclecount is always zero as the only code that modifies the variable > is under a condition that the variable is already non-zero while its initial > value is zero. > > That does not look right because there is some consequential code that it is > conditional on vn_alloc_cyclecount. > E.g.: > - vstir is never set; > - vn_alloc_hard never reaches 'goto alloc'. > > I checked compiled code just in case and it looks that the compiler agrees with > me: vn_alloc_cyclecount is optimized out as well as all code depending on it not > being zero. > > The vnode recycling code (with its many parameters and paths) has never been > easy for me to understand, so I am not sure about all implications but there is > one real-world problem that prompted me to look into the code. Not to be unnecessarily cryptic, let me describe the problem. vnode_list has hundreds of thousands of vnodes (close to desiredvnodes), all of them either being held (v_holdcnt > 0) or being already freed (dead vnops, v_holdcnt == VHOLD_NO_SMR). Then, there is a thread looping over the list in vnlru_free_impl, fruitlessly, while holding and never dropping vnode_list_mtx. Then, there is a multitude of other threads blocked on vnode_list_mtx in various places, mostly different locations in vn_alloc_hard. Among those threads is vnlru_proc thread which is blocked on vnode_list_mtx in vlrureclaim. There are a lot of vnodes whose use count is zero (held only by the name cache?) but vnlru_proc rarely gets a chance to run because of all other threads that eventually directly call into vnlru_free_impl and grab the lock for a long time. It does not help that vlrureclaim is super nice to others by periodically dropping the lock and yielding. So whatever freeable vnodes it creates get quickly consumed. Overall, there is a super long delay for each vnode allocation, so the system feels totally unresponsive. I may have misunderstood some details of the problem and the code, but hope that my description paints a good picture for those who know the code. I believe that vn_alloc_hard never reaching 'goto alloc' is a big part of the problem. -- Andriy Gapon