From nobody Wed Aug 24 08:37:14 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MCKGM1VhWz4ZdbV for ; Wed, 24 Aug 2022 08:37:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MCKGL6KmGz3W9y for ; Wed, 24 Aug 2022 08:37:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MCKGL5Mx8z12qM for ; Wed, 24 Aug 2022 08:37:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 27O8bE6I002488 for ; Wed, 24 Aug 2022 08:37:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 27O8bEYg002487 for bugs@FreeBSD.org; Wed, 24 Aug 2022 08:37:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 266013] Reported system memory decreases on large systems Date: Wed, 24 Aug 2022 08:37:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: thj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1661330234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=094P/zk4pR00tOyVjLPO1Mx7qc5jjVrJWrXjJecVfDk=; b=L/C8zOFXNk1qHBZ5xpJOdky/4QC0CaCiIJpuRJeZTnKpB8Wuv8Vnr+CbcvX8kGRvx8P9zB FLY+oWe/79At64V7XPXjRy6UJLIWRajH0PqDWcAmCaXnibt3CyAGJaF0MWw+Ctt7CNOaFy MAJ0GiPZ91yRNSbC3gLvpRfIUkVbTjltJJ/WWU88qmIRgZs7BReRcg8zZEAt5sG99epGn3 8uVQOdM/j10i6ksghHlcUK1vDHx2YoHWtdJL9Y0WtJSyyeVcZAi4ps/Y2KomYDp46YLa44 1PqGKdi+HZATI2RpGdoOI3FDRB36jSA0ZPG1xKeVwgDoIHXcTApNRKyM+nhtfw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1661330234; a=rsa-sha256; cv=none; b=ERnfPduQgwcM++bv0jiwg095bLzXyo8aVexSr0ypxkhlETgCJiZInNg9IpwQmGi6xvc/IY V919VgnbF8Jg8TG7uU0deNj+uDhwY6sUhcIESve66Toy86VD0pgc+5SyPyVFGAuwiDxsPH Plnd/6dkrL1pC4C/caW7BVl72a7ehAD3Lmfap3Wz2ljbQ00iK9BsnOM0tJUYL5uzvDLGzP nfwOdl028Cg4pwHvD1pNLbqCSpTb174n1qnKZwU0N/0hGdPVxPNaw0dbZebxLBQrbcXhdq 4/MZGUvtEojxLjGRgUJuSsjiMeyXOHHLgcGR7s4A4cmYNFBY029Q02rJKJhxsA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D266013 Bug ID: 266013 Summary: Reported system memory decreases on large systems Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: thj@FreeBSD.org Created attachment 236086 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D236086&action= =3Dedit Plot of memory reported by system On large database systems we have, the memory reported by top (as reported = by=20 the vm sysctls) decreases over time. This leads to the appearance that Free= BSD is loosing pages. We are experiencing this on a number of systems running FreeBSD 13, with memories in the Terrabyte ranges (systems have 4TB, 3.2TB), and multiple NU= MA domains (2 and 4). Over time the missing pages increase until they account = for ~50% of installed system memory with roughly 1.5TB missing on the 3.2TB machines. Over time the memory fields report by top (active, inactive, laundry, wired, buf, free) stopping summing up to the amount of memory in the machine.=20 Included is a plot showing the total memory that would be reported by top a= s a percentage of the total pages the system has. We have captured the output of `sysctl vm` on effected systems over long periods, an hour with 1 second sampling (upper plot) and 24 hours with 1 mi= nute sampling (lower two plots). While debugging this effect we patched the vm system to report the per doma= in page_count. The upper plot in the figure shows this information for each of= the memory domains in that system as well. These systems are part of a Galera MySQL cluster. When the Galera processes= are restarted the available memory in the system returns. --=20 You are receiving this mail because: You are the assignee for the bug.=