From nobody Mon Mar 04 14:42:57 2024 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpLyx4MfWz5CkL5 for ; Mon, 4 Mar 2024 14:43:05 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpLyx1QYkz42BB for ; Mon, 4 Mar 2024 14:43:05 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 424EgxBQ094620 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Mon, 4 Mar 2024 09:42:59 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:d96b:3a01:c295:f6b0] ([IPv6:2607:f3e0:0:4:d96b:3a01:c295:f6b0]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 424Egv8M081281 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Mon, 4 Mar 2024 09:42:57 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <7ee4c632-308a-4ecb-a386-0a9b74b57044@sentex.net> Date: Mon, 4 Mar 2024 09:42:57 -0500 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS ARC memory requirements Content-Language: en-US To: Andrea Venturoli , freebsd-questions@freebsd.org References: <7f7a9f1f-8a2b-41ef-a6a7-1dc1e2f160af@netfence.it> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <7f7a9f1f-8a2b-41ef-a6a7-1dc1e2f160af@netfence.it> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4TpLyx1QYkz42BB On 3/4/2024 2:19 AM, Andrea Venturoli wrote: > Hello. > > I'm planning a machine to be used (almost) only as a backup host. > It will have a "system" SSD-based ZFS pool (possibly 1TB mirror) for > OS and programs and a large dedicated HDD-based ZRAID1 for backups > (initially 160-200 TB, but may grow in the future). > > my guess would be that I don't need that much RAM and 64 or even 32GiB > would be enough. > 64G should be fine. I have a backup server with 102TB across 3 raidZ1 vdevs, CPU E3-1230 v6 @ 3.50GHz with LSI MegaRAID SAS 9272-8i controller with JBOD disks on a SAS2 backplane.  Its mostly a sink for backup data and occasionally I will restore something from backups. The RAM would help somewhat at restore time, but I get the sense it will be diminishing returns especially if you are restoring things one at a time or sequentially. If I read a big file (that is not in cache) and dump it to /dev/null I get acceptable (for me) throughput  dd if=netflow.2024.02.07.20.00 bs=1m of=/dev/null status=progress   18360565760 bytes (18 GB, 17 GiB) transferred 24.003s, 765 MB/s 17986+1 records in 17986+1 records out 18859928380 bytes transferred in 24.643516 secs (765309973 bytes/sec) I imagine spreading it across more vdevs and using faster HDDs will improve speeds as these are all average grade HDDs but the read speeds meet my needs for now. Here are my arc stats FWIW before and after that big read. ------------------------------------------------------------------------ ARC Summary: (HEALTHY)         Memory Throttle Count:                  0 ARC Misc:         Deleted:                                4.20    b         Mutex Misses:                           1.92    m         Evict Skips:                            136.33  k ARC Size:                               84.76%  53.21   GiB         Target Size: (Adaptive)         84.78%  53.22   GiB         Min Size (Hard Limit):          3.17%   1.99    GiB         Max Size (High Water):          31:1    62.78   GiB         Compressed Data Size:                   49.37   GiB         Decompressed Data Size:                 54.13   GiB         Compression Factor:                     1.10 ARC Size Breakdown:         Recently Used Cache Size:       52.50%  27.94   GiB         Frequently Used Cache Size:     47.50%  25.28   GiB ARC Hash Breakdown:         Elements Max:                           11.21   m         Elements Current:               65.86%  7.38    m         Collisions:                             2.69    b         Chain Max:                              12         Chains:                                 1.85    m ------------------------------------------------------------------------ ARC Efficiency:                                 60.98   b         Cache Hit Ratio:                95.92%  58.49   b         Cache Miss Ratio:               4.08%   2.49    b         Actual Hit Ratio:               95.63%  58.31   b         Data Demand Efficiency:         76.97%  1.60    b         Data Prefetch Efficiency:       6.04%   2.20    b         CACHE HITS BY CACHE LIST:           Anonymously Used:             0.24%   141.86  m           Most Recently Used:           6.58%   3.85    b           Most Frequently Used:         93.13%  54.47   b           Most Recently Used Ghost:     0.03%   17.37   m           Most Frequently Used Ghost:   0.02%   12.78   m         CACHE HITS BY DATA TYPE:           Demand Data:                  2.10%   1.23    b           Prefetch Data:                0.23%   133.14  m           Demand Metadata:              97.59%  57.08   b           Prefetch Metadata:            0.08%   46.43   m         CACHE MISSES BY DATA TYPE:           Demand Data:                  14.78%  368.09  m           Prefetch Data:                83.12%  2.07    b           Demand Metadata:              1.52%   37.87   m           Prefetch Metadata:            0.58%   14.47   m --------------------------- ARC Summary: (HEALTHY)         Memory Throttle Count:                  0 ARC Misc:         Deleted:                                4.20    b         Mutex Misses:                           1.93    m         Evict Skips:                            136.40  k ARC Size:                               84.79%  53.23   GiB         Target Size: (Adaptive)         84.78%  53.22   GiB         Min Size (Hard Limit):          3.17%   1.99    GiB         Max Size (High Water):          31:1    62.78   GiB         Compressed Data Size:                   49.54   GiB         Decompressed Data Size:                 61.72   GiB         Compression Factor:                     1.25 ARC Size Breakdown:         Recently Used Cache Size:       52.44%  27.91   GiB         Frequently Used Cache Size:     47.56%  25.31   GiB ARC Hash Breakdown:         Elements Max:                           11.21   m         Elements Current:               59.84%  6.71    m         Collisions:                             2.69    b         Chain Max:                              12         Chains:                                 1.60    m ------------------------------------------------------------------------ ARC Efficiency:                                 60.99   b         Cache Hit Ratio:                95.92%  58.50   b         Cache Miss Ratio:               4.08%   2.49    b         Actual Hit Ratio:               95.63%  58.32   b         Data Demand Efficiency:         76.97%  1.60    b         Data Prefetch Efficiency:       6.04%   2.20    b         CACHE HITS BY CACHE LIST:           Anonymously Used:             0.24%   141.88  m           Most Recently Used:           6.58%   3.85    b           Most Frequently Used:         93.13%  54.48   b           Most Recently Used Ghost:     0.03%   17.37   m           Most Frequently Used Ghost:   0.02%   12.79   m         CACHE HITS BY DATA TYPE:           Demand Data:                  2.10%   1.23    b           Prefetch Data:                0.23%   133.16  m           Demand Metadata:              97.59%  57.09   b           Prefetch Metadata:            0.08%   46.43   m         CACHE MISSES BY DATA TYPE:           Demand Data:                  14.78%  368.10  m           Prefetch Data:                83.12%  2.07    b           Demand Metadata:              1.52%   37.88   m           Prefetch Metadata:            0.58%   14.47   m     ---Mike