From nobody Tue Aug 23 18:48:35 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MBytS34fvz4bGW6 for ; Tue, 23 Aug 2022 18:48:48 +0000 (UTC) (envelope-from porsolic@gmail.com) Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MBytR3yCNz3PhJ for ; Tue, 23 Aug 2022 18:48:47 +0000 (UTC) (envelope-from porsolic@gmail.com) Received: by mail-io1-xd2a.google.com with SMTP id n202so6122394iod.6 for ; Tue, 23 Aug 2022 11:48:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc; bh=xye1pfmn2ckMy8gJW5zIsYMtf9+GGHHdU48HOLlQNJE=; b=MD4tD0KcbX6FAqAeGjlRyIhgYgQO7cYripQsQrS4zviceASp+5oMskVK097MvsRTUI XETnPAHExtpHV8Lh++m8e8LxNp0+v6Y7GS49caBR5+AbHz4QqM81ZtjhiG9S2v9F10lG MqVeFGH06/9ciSuEvPYUpn8hpqslFTK+toBNCHtV6LfOiMBy8TlhFJ1Mjf9JK5UcCchQ iZrWE19ZkMkR6r8uQuCyy5C/fsjc7l+q0kYhJv6LmhDJ/EB2qDJpUelfckgZOY8fThbC GsnftV60BmwRCYoOh33W0L97xLADTXl26RodXvXrk+zCgsVSzF85bUkYXi0U7DLyJiKz nN2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc; bh=xye1pfmn2ckMy8gJW5zIsYMtf9+GGHHdU48HOLlQNJE=; b=JNwDE+Lp36Sw4WlM7P5jOBMnOzdp7VQggDiufmnXQKobTYPHDk9fRs480lCPmnZjZM qIBdTBwgVIbt+yIU8CnLRN2xTB764XEc9F4TnJeWYoC6taPVj7CcnNRqI3rnUjP2ctjs FjeJ+918dPLyZ9EQA0zKCdTtbnQP8ECdeFA0htIXWhXgJOaZyDiNBQdIfMzc27JdwNr+ D+Gmgs/oeyeFIyUjIQnIuAoLOKGOhyc47UDo1UMREepOs6i1nzko/qp5GqDpVW6FiNg3 DxMgdvOSKdAYZasHWCeVxpH7bnxNw4yOC+jzprwEAnGrm++b1ltLX6XwNooQaV0jewjI p+hA== X-Gm-Message-State: ACgBeo06aqcoLQRrHH63AeCMN2zoMwcJb58Cn6/d6MHrF2L1tEI99/9n Jdj5Ytbi7PUjLlk5patsVbhEHV9C1/0YxLZfo6Kxarr1TvU0+A== X-Google-Smtp-Source: AA6agR4egPI9CRzzDokLfp6pv0czCea5dXZCf9S0WDT0+ZaSl5VxhBRx4ZyHDejJaX7orgI9Lgtgh6fD2XxKLdoJFtI= X-Received: by 2002:a05:6638:2386:b0:344:c9ed:a48e with SMTP id q6-20020a056638238600b00344c9eda48emr12692181jat.180.1661280526531; Tue, 23 Aug 2022 11:48:46 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: porsolic Date: Tue, 23 Aug 2022 20:48:35 +0200 Message-ID: Subject: Too much RAM used for wired memory To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000a8049a05e6ed03c5" X-Rspamd-Queue-Id: 4MBytR3yCNz3PhJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=MD4tD0Kc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of porsolic@gmail.com designates 2607:f8b0:4864:20::d2a as permitted sender) smtp.mailfrom=porsolic@gmail.com X-Spamd-Result: default: False [-3.95 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.95)[-0.952]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2a:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --000000000000a8049a05e6ed03c5 Content-Type: text/plain; charset="UTF-8" My PC uses too much wired memory after few days. After two weeks of uptime, wired memory usage is more than 50% of my RAM (17G Wired, 32 GB of physical RAM) PC is used with ZFS, 1 VM (bhyve, 1 CPU, 256 MB of RAM, passthru PCIe LAN card, pinned to 1 phy CPU, priority/nice is -10), Xorg, multiple Firefox instances (one used for playing videos), work in shell (zsh, tmux, vim, ...), GUI filemanager, GUI PDF reader and sometimes with KiCAD and/or WINE. Even after stopping VM, all programs, Xorg, tmux, dbus, avahi, moused, cups, other daemons, exporting non-root ZFS pool, going to single user mode - wired memory still doesn't decrease man vmstat shows: -z Report on memory used by the kernel zone allocator, uma(9), by zone. Is that the same as wired memory? How can I see what exactly is using that much of wired memory? What metrics should I use to pinpoint issue? vmstat? sysctl counters? # uname -srpKU FreeBSD 13.1-RELEASE amd64 1301000 1301000 # top -nSz | head -6 last pid: 1884; load averages: 2.57, 2.96, 2.91 up 15+05:21:55 15:00:00 191 processes: 5 running, 184 sleeping, 1 zombie, 1 waiting CPU: 5.1% user, 0.0% nice, 4.2% system, 0.1% interrupt, 90.7% idle Mem: 6136M Active, 3218M Inact, 1986M Laundry, 17G Wired, 96K Buf, 719M Free ARC: 1513M Total, 548M MFU, 419M MRU, 11M Anon, 17M Header, 517M Other 464M Compressed, 633M Uncompressed, 1.36:1 Ratio # vmstat -s | grep wired 4354572 pages wired down 65537 virtual user pages wired down # vmstat -m | (head -n1 && sort -hk3 | tail -5) Type InUse MemUse Requests Size(s) vtbuf 24 4560K 70 4096 linux 52226 17342K 2088029157 16,32,64,128,256,384,512,1024,2048,4096,8192,16384,32768,65536 amdvi 16924 67689K 16924 32,4096 solaris 888982 303053K 5679401089 16,32,64,128,256,384,512,1024,2048,4096,8192,16384,32768,65536 drm_kms 18446744073709504758 18014398509470267K 14 64 # vmstat -z | sort -t , -k3 | (head -n1 && tail -5) ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN malloc-384: 384, 0, 368738, 1982,1477234413, 0, 0, 0 dnode_t: 816, 0, 383683, 192,235818160, 0, 0, 0 dmu_buf_impl_t: 296, 0, 386372, 27938,569079198, 0, 0, 0 vm pgcache: 4096, 0, 646128, 1726,2444497080,66010, 0, 0 vm pgcache: 4096, 0, 6560096, 4312,6998441782,53699, 0, 0 # zfs-stats -MA ------------------------------------------------------------------------ ZFS Subsystem Report Tue Aug 23 15:00:00 2022 ------------------------------------------------------------------------ System Memory: 20.73% 5.99 GiB Active, 10.87% 3.14 GiB Inact 57.45% 16.61 GiB Wired, 0.00% 0 Bytes Cache 2.40% 709.61 MiB Free, 8.56% 2.47 GiB Gap Real Installed: 32.00 GiB Real Available: 92.97% 29.75 GiB Real Managed: 97.19% 28.92 GiB Logical Total: 32.00 GiB Logical Used: 88.01% 28.16 GiB Logical Free: 11.99% 3.84 GiB Kernel Memory: 16.00 EiB Data: 100.00% 16.00 EiB Text: 0.00% 49.03 MiB Kernel Memory Map: 4.00 GiB Size: 63.11% 2.52 GiB Free: 36.89% 1.48 GiB ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 39.87 m Mutex Misses: 7.69 m Evict Skips: 3.07 b ARC Size: 36.95% 1.48 GiB Target Size: (Adaptive) 28.12% 1.12 GiB Min Size (Hard Limit): 23.24% 952.04 MiB Max Size (High Water): 4:1 4.00 GiB Compressed Data Size: 464.17 MiB Decompressed Data Size: 632.91 MiB Compression Factor: 1.36 ARC Size Breakdown: Recently Used Cache Size: 55.47% 839.54 MiB Frequently Used Cache Size: 44.53% 673.85 MiB ARC Hash Breakdown: Elements Max: 448.86 k Elements Current: 12.85% 57.67 k Collisions: 2.04 m Chain Max: 5 Chains: 408 ------------------------------------------------------------------------ --000000000000a8049a05e6ed03c5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
My PC uses too much wired memory after few days.
After = two weeks of uptime, wired memory usage is more than 50% of my RAM (17G Wir= ed, 32 GB of physical RAM)
PC is used with ZFS, 1 VM (bhyve, 1 CPU, 256 = MB of RAM, passthru PCIe LAN card, pinned to 1 phy CPU, priority/nice is -1= 0), Xorg, multiple Firefox instances (one used for playing videos), work in= shell (zsh, tmux, vim, ...), GUI filemanager, GUI PDF reader and sometimes= with KiCAD and/or WINE.

Even after stopping VM, all programs, Xorg,= tmux, dbus, avahi, moused, cups, other daemons, exporting non-root ZFS poo= l, going to single user mode - wired memory still doesn't decrease
<= br>man vmstat shows:
-z =C2=A0 =C2=A0 =C2=A0Report on memory used by the= kernel zone allocator, uma(9), by zone.

Is that the same as wired m= emory?

How can I see what exactly is using that much of wired memory= ?
What metrics should I use to pinpoint issue? vmstat? sysctl counters?<= br>
# uname -srpKU
FreeBSD 13.1-RELEASE amd64 1301000 1301000

= # top -nSz | head -6
last pid: =C2=A01884; =C2=A0load averages: =C2=A02.= 57, =C2=A02.96, =C2=A02.91 =C2=A0up 15+05:21:55 =C2=A0 =C2=A015:00:00
19= 1 processes: 5 running, 184 sleeping, 1 zombie, 1 waiting
CPU: =C2=A05.1= % user, =C2=A00.0% nice, =C2=A04.2% system, =C2=A00.1% interrupt, 90.7% idl= e
Mem: 6136M Active, 3218M Inact, 1986M Laundry, 17G Wired, 96K Buf, 719= M Free
ARC: 1513M Total, 548M MFU, 419M MRU, 11M Anon, 17M Header, 517M = Other
=C2=A0 =C2=A0 =C2=A0464M Compressed, 633M Uncompressed, 1.36:1 Rat= io

# vmstat -s | grep wired
=C2=A0 4354572 pages wired down
= =C2=A0 =C2=A0 65537 virtual user pages wired down

# vmstat -m | (hea= d -n1 && sort -hk3 | tail -5)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Type InUse MemUse Requests =C2=A0Size(s)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vtb= uf =C2=A0 =C2=A024 =C2=A04560K =C2=A0 =C2=A0 =C2=A0 70 =C2=A04096
=C2=A0= =C2=A0 =C2=A0 =C2=A0 linux 52226 17342K 2088029157 =C2=A016,32,64,128,256,= 384,512,1024,2048,4096,8192,16384,32768,65536
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 amdvi 16924 67689K =C2=A0 =C2=A016924 =C2=A032,4096
=C2=A0 =C2=A0 = =C2=A0 solaris 888982 303053K 5679401089 =C2=A016,32,64,128,256,384,512,102= 4,2048,4096,8192,16384,32768,65536
=C2=A0 =C2=A0 =C2=A0 drm_kms 18446744= 073709504758 18014398509470267K =C2=A0 =C2=A0 =C2=A0 14 =C2=A064

# v= mstat -z | sort -t , -k3 | (head -n1 && tail -5)
ITEM =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SIZE =C2=A0LIMIT =C2= =A0 =C2=A0 USED =C2=A0 =C2=A0 FREE =C2=A0 =C2=A0 =C2=A0REQ =C2=A0 =C2=A0 FA= ILSLEEP XDOMAIN
malloc-384: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 38= 4, =C2=A0 =C2=A0 =C2=A00, =C2=A0368738, =C2=A0 =C2=A01982,1477234413, =C2= =A0 0, =C2=A0 0, =C2=A0 0
dnode_t: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0816, =C2=A0 =C2=A0 =C2=A00, =C2=A0383683, =C2=A0 =C2=A0= 192,235818160, =C2=A0 0, =C2=A0 0, =C2=A0 0
dmu_buf_impl_t: =C2=A0 =C2= =A0 =C2=A0 =C2=A0 296, =C2=A0 =C2=A0 =C2=A00, =C2=A0386372, =C2=A0 27938,56= 9079198, =C2=A0 0, =C2=A0 0, =C2=A0 0
vm pgcache: =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A04096, =C2=A0 =C2=A0 =C2=A00, =C2=A0646128, =C2=A0 =C2= =A01726,2444497080,66010, =C2=A0 0, =C2=A0 0
vm pgcache: =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A04096, =C2=A0 =C2=A0 =C2=A00, 6560096, =C2=A0 =C2= =A04312,6998441782,53699, =C2=A0 0, =C2=A0 0

# zfs-stats -MA

= ------------------------------------------------------------------------ZFS Subsystem Report Tue Aug 23 15:00:00 2022
----------------------= --------------------------------------------------

System Memory:
20.73% 5.99 GiB Active, 10.87% 3.14 GiB Inact
57.45% 16.61 GiB Wi= red, 0.00% 0 Bytes Cache
2.40% 709.61 MiB Free, 8.56% 2.47 GiB Gap
<= br> Real Installed: 32.00 GiB
Real Available: 92.97% 29.75 GiB
= Real Managed: 97.19% 28.92 GiB

Logical Total: 32.00 GiB
L= ogical Used: 88.01% 28.16 GiB
Logical Free: 11.99% 3.84 GiB

= Kernel Memory: 16.00 EiB
Data: 100.00% 16.00 EiB
Text: 0.= 00% 49.03 MiB

Kernel Memory Map: 4.00 GiB
Size: 63.11% 2.5= 2 GiB
Free: 36.89% 1.48 GiB

---------------------------------= ---------------------------------------

ARC Summary: (HEALTHY)
M= emory Throttle Count: 0

ARC Misc:
Deleted: 39.87 m
Mute= x Misses: 7.69 m
Evict Skips: 3.07 b

ARC Size: 36.95% 1= .48 GiB
Target Size: (Adaptive) 28.12% 1.12 GiB
Min Size (Hard Lim= it): 23.24% 952.04 MiB
Max Size (High Water): 4:1 4.00 GiB
Compre= ssed Data Size: 464.17 MiB
Decompressed Data Size: 632.91 MiB
C= ompression Factor: 1.36

ARC Size Breakdown:
Recently Used Cach= e Size: 55.47% 839.54 MiB
Frequently Used Cache Size: 44.53% 673.85 MiB=

ARC Hash Breakdown:
Elements Max: 448.86 k
Elements Curr= ent: 12.85% 57.67 k
Collisions: 2.04 m
Chain Max: 5
Chai= ns: 408

--------------------------------------------------------= ----------------
--000000000000a8049a05e6ed03c5--