From nobody Sat Feb 04 15:16:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P8GM765TBz3nW5l for ; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) (envelope-from debdrup@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P8GM75fRcz3wNJ for ; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) (envelope-from debdrup@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675523779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ZXeHEBriOVyVhW83egLaK4xy3Jmz97FPB6mDYgWH8RE=; b=E/Wuo+PgdtjZ13HoYGqTMzg+yRS8yTq0CRdwG7HeHWtGidgKcVvydMk0RbPf+Pw70kbGBj EpNNfx6IYQdkanYcWc3YTnGyWyufeTgcINMIMgmWcq2qiSgFEBopdnNvHyYYJQa+Zymnoo 8FIu6t3jXZjUXZlyDVQS9274HEka0SMMa2Y5ZAmwzx7xtf5xHDng7QMtG8NG4Wi8rOpeEe Li6FQDxMFQyp8Qlgyf46WTfUz7xg+d7vPXHzzuNBdbG5Rjqj4N9ljCMDGRuANoPKaXpINM d4LWQeWMP20ZZt3irSYvBOThs23MuBjt/w95cBXPu9pld7pOcolgerbEVkcOFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675523779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ZXeHEBriOVyVhW83egLaK4xy3Jmz97FPB6mDYgWH8RE=; b=GUXS6p5AQmRpo2VOCmkZ5OiXEu7jnvxzP21T/bIFyiWwi5kKpuXdxKWK7+NU9wOMgm6bfz TQ82rBHtCxkvQHL8aRguMGw3B5BnC9KFtAFk3wl/uzxFvOyaGYTSMMEe+P1P34IIEb5WlO PYHQ2PZnNjSs4uuboNDbdmYnkmUTUkdVhIjVcikxtOSRwe5LtqebdzHCmpJ/4GivxcGzEd xgr6imEblCdRU4B+UIuQIy6VB+/sqoRlklNxmDOhtkum+59zcUMJTpoN+UcNzGbYzthzka T2dTJpDRH3RZdPuN/JYVIxLsDX5shTVus5JqbcyFV6IHN4IOb+8h2NB4PG664w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675523779; a=rsa-sha256; cv=none; b=A/FyxZf92aAAqF7j+16gTn5a4JinhfkYrNaT+I5HcUQeF4TrCqCtEVIK/gf0zRLPvwMJRu HqGD5vS9o5PidiOsqLsJAlEe1Y5YOHURCXNfK7XQhro+YdBbnKLeiUDNM3kXKs9cvSz/tX tCPwgwQilllLvLhptfQv0cV264ZtOxIc8sy2ReDX/Lwd9MGQOa+x291nu+AvGyhIjx3J76 okcY0shNAxsuNJz+7mi3BU2eqlK0fMPdYrrlUA44iXDFIYmzk78MzbLCXru97R75TbnYB3 ywOKlwkLRmIAw4/O4HDNLIX5Cab6fzkTF0GudmlfC3s+k3vd3eq8kA4O7M4adg== Received: by freefall.freebsd.org (Postfix, from userid 1471) id B0458122A3; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) Date: Sat, 4 Feb 2023 16:16:17 +0100 From: Daniel Ebdrup Jensen To: freebsd-hackers@freebsd.org Subject: Re: Swap, ZFS & ARC Message-ID: <20230204151617.wlan22x5jvae5jzm@geroi.local> References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="n5tann3f7es4la3y" Content-Disposition: inline In-Reply-To: X-ThisMailContainsUnwantedMimeParts: N --n5tann3f7es4la3y Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline On Thu, Feb 02, 2023 at 02:28:48PM +0000, jbo@insane.engineer wrote: >Hello folks, > >Based on a discussion on the forums not so long ago I tried to figure out how swap usage on a ZFS system plays together with ARC. However, I could find very little to no information on this which leads me to believe that there is some "core concept" I might be oblivious to. > >The main question is basically this: Your system starts to swap out data from RAM to your swap partition. This swap data on disk ultimately resides somewhere in a ZFS pool. If this data then gets accessed, it might be cached by ARC essentially eating up memory again which seems counter productive. >Is there any magic which prevents swap partitions from being loaded into ARC? Or is this a non-issue for some other reason? > >Best regards, >~ joel Hi Joel, The catch-22 mentioned elsewhere in the thread isn't unique, it's a function of how virtual memory systems work when they have paging, even if they're designed to try and work around them (which, as others have pointed out, is how FreeBSD works). The only way around it that I know of is to adjust the VM watermarks that control when certain triggers happen, but unfortunately this process is workload dependent, so there's no real way of making recommendations short of "if you're still seeing the catch-22 after adjusting the values, adjust them more". The values that need to be adjusted are the following OIDs, using sysctl(8): vm.v_free_min, vm.v_free_target, vm.v_free_reserved, vm.v_inactive_target, and vm.v_free_severe. There is a pretty big downside to this, though - which is that it means a much bigger portion of your memory will be completely unused at any given point; which consequently mean your ARC is much less efficient, there's less room for more things in other caches, and you're still paying for the electricity to keep the memory state of the unused memory. There's also an issue you can run into if you ever have a panic(9) and it's caused by something in ZFS, because that'll mean that the core file can't be reliably expected to produce a meaningful backtrace. Ultimately, it's a question of what's easier - if you've setup your pool with no swap, and didn't reserve some space for future expansion and/or can't be arsed to do the zfs send | receive onto a new pool that has the space for it, the above method will at least let you get swap on ZFS without having to worry about the ZPL from using file extents (which is never a good idea, not even on UFS). As for your second question, the dataset zfsprops(7) parameter that you're looking for is primarycache, as it can be set to none. You'll also want to use org.freebsd:swap, turn off checksumming and compression, and disable sync. Yours, Daniel Ebdrup Jensen --n5tann3f7es4la3y Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAABCgB9FiEEDonNJPbg/JLIMoS6Ps5hSHzN87oFAmPedsFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDBF ODlDRDI0RjZFMEZDOTJDODMyODRCQTNFQ0U2MTQ4N0NDREYzQkEACgkQPs5hSHzN 87oEIAf/aXiuKuytzfTqcLUaVppewhFx1DFM2jYpPCW5LQNGEHlYP7nEwzoMLhN8 ACBLer8H//d9C2x1znTGRmeEhtnOwnpyEu1TInbRhiUJ21DufDzQ5OHzTI7ztxWk eVde3WtcZ0mdEShDpCUASAkRexA18ycd/t5YXqjgsik8TiZOfVVDMLvinxmPBelv /UC9gX8cdUnr2hF8e3UytP9JTpnAVahFvlxtTsiHEKk4ViPaTWwmakMR/npk7czW ff1omhndZt/RKvJOhVA/myV1TugNYo1HfRGj5QPlSMbsum2B0CM4oItaTEs1Ava2 eby1q2V24xSavDJnAqa9YA6eesFMhg== =P653 -----END PGP SIGNATURE----- --n5tann3f7es4la3y--