From nobody Wed Aug 07 15:20:25 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WfDQ25Drsz5SPJ3 for ; Wed, 07 Aug 2024 15:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WfDQ21Qd4z4kYL for ; Wed, 7 Aug 2024 15:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1723044026; a=rsa-sha256; cv=none; b=o1g4FQo0y7irLQifoow83DKmCLrIRmu2wbwhFDxLusPjh9l7Ioxgg/7tcOd2Xw8B4KOely ZSuFiKoNhjM9jhvEYm3jYzUW9gyj0+0Ia9FbejnelPehsU9ActNPH38GGr36K3Yqf7aEgk z3fl2S2Vbd3wpLafGxN/FbkC45uuba83ybbBxfT4sjQBwvr/fwqRg3V+VgO9Ke7BjhZakW k4EC/8KhC37taKux4hrnsNua2NSKcFTiHiMC6kZY+SjpYsZlnNVF836g8f/NZEEtYM31pN 5iF7phT+QH1jVx7cEMteOYmS3UsoveF1p5wN9QEYKE5Cgb7iZgu6YSULd/rlaA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1723044026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Xhw0+qyP1oGNa7+BEmvyuhGLBdOAchSsucHwh8XiDhw=; b=rVTa1qU+sovd1cJEYCuOqrHmFrcOwY9VQMcFjT3VkGv8zl3G4JyIVKRD51KCpYCNG7YP5J Se23bApgQmV850lBOSIEcPLpciQl/4yd0uAaA1UZ1YQAiDDDdHthFDcl5+C4WbTucRiaXL jHULVC+dbQGgwhJfS8gb+Lo9mRBeu29I1AH6KqjVLRoQYtF22Vnwxm880mbfIbyz9xd1dN OUxBL6VQeFaNEmTaQnkNJRquQ+jNtHQW23cNPrEdcqahQWl1tydneFeAxfVSTQHeDH1MGe CUDVxCVdEVQh0FqKHVbmCljkSTBHJcTTi7t8nj++kd5zXzf8P4e8GPR/kyEGAA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WfDQ20PkbzwDG for ; Wed, 7 Aug 2024 15:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 477FKPYa015137 for ; Wed, 7 Aug 2024 15:20:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 477FKPjf015136 for bugs@FreeBSD.org; Wed, 7 Aug 2024 15:20:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 280671] Memory leak on FreeBSD 13.3 and 14.1 Date: Wed, 07 Aug 2024 15:20:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sre@truespeed.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280671 Bug ID: 280671 Summary: Memory leak on FreeBSD 13.3 and 14.1 Product: Base System Version: 14.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: sre@truespeed.com Created attachment 252589 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D252589&action= =3Dedit chart of memory usage Good afternoon We recently upgraded the operating system for one of our servers from FreeB= SD 12.4 to 13.3. This server uses the Generic Kernel, a Mirrored ZFS Zpool, and has a few Ja= ils on FreeBSD 12.0 that are running standard applications (Java based web services, PostgreSQL, RabbitMQ). The server has 96GB of RAM and was not experiencing memory shortages prior = to upgrading.=20 We followed the standard upgrade process as described here: https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgr= ade=20 Followed by upgrading packages and our zpools. We have not upgraded our Jai= ls. After upgrading, we began to experience what seemed like a memory leak on t= he server. Over time the Inactive Memory would grow before dumping gigabytes a= t a time into Laundry that was never cleaned before eventually running out of f= ree Memory and begin thrashing. At this point we lose access to the server, and= the services it is running become unresponsive. We resolve this by power cycling the server, and it returns to normal use on reboot. We are currently rebooting the server every few days before it enters the thrashing state, but this is not a feasible long-term solution, and we beli= eve there is a Memory Leak that is causing this situation. As part of debugging the memory issue, we have tried to recover memory by turning off existing jails (as shown on the chart below by a large dip in memory usage around 1am), but this memory is rapidly consumed again. Also, = when the server is close to entering the thrashing state, we have turned off eve= ry jail and service (except for a few critical ones, ie SSHd) to see how much memory is being =E2=80=9Clost=E2=80=9D, and it was about 39GB~, with 9GB us= ed by ARC. Mem: 63M Active, 8656M Inact, 18G Laundry, 17G Wired, 50G Free Also, we have limited ARC usage with the following sysctl vfs.zfs.arc_max, = but that hasn=E2=80=99t made any meaningful impact. There is nothing else standing out on the server, no unusual CPU utilisatio= n, no unusual network traffic, all the crons are as before the upgrades, and we haven=E2=80=99t deployed any additional jails or services to the server. We then upgraded from 13.3 to 14.1 as there was a ZFS Memory Leak Errata in= the 14.1 release notes: https://www.freebsd.org/security/advisories/FreeBSD-EN-24:10.zfs.asc, but t= hat hasn=E2=80=99t resolved our issue. As you can see from the charts below this is the memory usage pattern we are dealing with, this data is being pulled from sysctl by the node_exporter for Prometheus. The attached chart_1 shows the memory usage of the server over the last wee= k. The attached chart_2 shows the final hour before the server begins thrashin= g. Laundry grows to 62GB and Inactive and Free Memory are both reduced to <1GB. The available swap is 4GB but it does not seem like it=E2=80=99s getting us= ed to justify needing to increase the swap space. We have also temporarily disabl= ed the SWAP entirely and that hasn=E2=80=99t made any difference. If you require any additional information, please let us know. Kind regards, Truespeed --=20 You are receiving this mail because: You are the assignee for the bug.=