From nobody Wed Jul 19 22:11:30 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R5qmZ3ndmz4mxwm; Wed, 19 Jul 2023 22:11:58 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R5qmY6bSLz3tjw; Wed, 19 Jul 2023 22:11:57 +0000 (UTC) (envelope-from scott.gasch@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-26586e824e7so44792a91.3; Wed, 19 Jul 2023 15:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689804716; x=1692396716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PhDbvCfOUCmF25V5xf674+P9I+zo/hOrmqkPpvpR9g0=; b=fLFVXEwL6LlggX/44TOL2HmXSSy9acrg/J/mGaDbrNqWsrazMYpjz8sKKWWmBd6q6E mJgHFfhuoTdEn0gEQ8lBzxlFXxg6Fz4mNH7LqMM+OelAcESoTZ57UMs5HcQPSwahgbSl elUYX4ywTE2GYQ4zWEJYlEW18op0JV8WZL233+KL5zrqNZGovKYlZSHOJQznhUf5yUYt g3UVyGkep6icQV6b9089tgx+MnfZxwywEkAfpVpmNV8mesLfzQz8JAtycx7X+MBI1sfA cPONpnA6rYau12tQ3KTTHcUwBCfg4fk9CbY+pH+rtt3KcOVEJV+Y4v2z9UdPDGYyTcu0 S1lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689804716; x=1692396716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PhDbvCfOUCmF25V5xf674+P9I+zo/hOrmqkPpvpR9g0=; b=QGmtevLi6Z0lhuf8w0alBYu37x1+wothjw9LIV7YM/RWD7HEA+ukTuJ9F4dE6MipAS +v2MDGs0qEXnWotXmZ7T8WStJUpum6Su7enI6mNGGL+yp5XZeS8q7/VXd2/beow5uKWJ wYMiu4f7Muk6IKbkIkdKkhGTZJ7DtGW4AlOGjGbp7c6q8DH0n3zjtlv7KmbwUZpFBfHP g4iqVzvY53VQ1IXHYtRAAIcT+7ZFH98csh0thJjKqj3/LhZTQZrh1pVv6ZUaSehSy4M6 1lTF2zgJa26IxZy8I5qTq3zqcbnKolZHkPfJuiINk9o56TZS0mElEa/4K1zc3tSynrqA c/qQ== X-Gm-Message-State: ABy/qLY/OlT2YGOwlFTdeHLKwX9+OoTCAv+sGC6fb60c/ZfgPLHuzmaf eUhBTYByss13Gk5Z/dpOv1tbhdWZWKsl+6yvhxEGqalfgXg= X-Google-Smtp-Source: APBJJlEorU3/q8U7SmRFvB+0RYXi2QoEKT6DB3cSWxIJ+cNbHo98NG1xdErXqgHau71TysDT3fJJIPm3RzEsG4LyigE= X-Received: by 2002:a17:90b:50b:b0:263:1af0:5541 with SMTP id r11-20020a17090b050b00b002631af05541mr2921306pjz.32.1689804716167; Wed, 19 Jul 2023 15:11:56 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Scott Gasch Date: Wed, 19 Jul 2023 15:11:30 -0700 Message-ID: Subject: Re: Swap filling up, usermode process swap usage doesn't explain To: Pete Wright Cc: freebsd-questions , freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000d8dc6f0600de5111" X-Rspamd-Queue-Id: 4R5qmY6bSLz3tjw X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --000000000000d8dc6f0600de5111 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, I'm using ZFS. Here's what top says: last pid: 88926; load averages: 1.20, 0.96, 0.87 up 5+17:48:34 15:09:58 274 processes: 1 running, 272 sleeping, 1 zombie CPU: 1.8% user, 0.0% nice, 0.5% system, 0.0% interrupt, 97.8% idle Mem: 1844M Active, 7777M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M Fre= e ARC: 24G Total, 2878M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other 18G Compressed, 25G Uncompressed, 1.33:1 Ratio Swap: 144G Total, 11G Used, 133G Free, 7% Inuse If I leave this alone it will grow to consume all available swap space. I'll try your fix with the sysctl knob and see what happens... I hope this is it, I've been fighting this for a while now. Thx, Scott On Wed, Jul 19, 2023 at 11:34=E2=80=AFAM Pete Wright = wrote: > > > On 7/19/23 07:49, Scott Gasch wrote: > > I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where, > > after about 10 days of uptime, my swap begins to fill up. > > > > > > > At least they agree about it being 11G. Is this kernel memory being > > paged out to swap? The machine has 128G of physical memory and isn't > > under very heavy load at the moment. > > > > Are you running ZFS by any chance? If so its possible it is trying to > use as much memory as possible for the ARC. I've seen this on a few > systems which lots of memory. One way to tell is to run "top" and look > at the ARC stats: > > last pid: 71322; load averages: 1.02, 0.94, 0.87 up > 8+18:38:34 11:31:26 > 376 processes: 1 running, 146 sleeping, 229 zombie > CPU: 0.6% user, 0.0% nice, 6.5% system, 0.0% interrupt, 93.0% idle > Mem: 3599M Active, 18G Inact, 4132M Laundry, 4272M Wired, 892M Free > ARC: 1749M Total, 651M MFU, 239M MRU, 1864K Anon, 13M Header, 844M Other > 216M Compressed, 758M Uncompressed, 3.52:1 Ratio > > > > On a few of my larger memory systems I cap the ARC by setting this > sysctl knob (this is like 45G on my system): > vfs.zfs.arc.max=3D45000000000 > > > -pete > > > -- > Pete Wright > pete@nomadlogic.org > @nomadlogicLA > --000000000000d8dc6f0600de5111 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Yes, I'm using ZFS.=C2=A0 Here's what top says:
last pid: 88926; =C2=A0load averages: =C2=A01.20, =C2=A00.= 96, =C2=A00.87 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0up 5+17:48:34 =C2=A015:09:58
274 processes: 1 ru= nning, 272 sleeping, 1 zombie
CPU: =C2=A01.8% user, =C2=A00.0% nice, =C2= =A00.5% system, =C2=A00.0% interrupt, 97.8% idle
Mem: 1844M Active, 7777= M Inact, 77G Laundry, 35G Wired, 750M Buf, 3367M Free
ARC: 24G Total, 28= 78M MFU, 18G MRU, 21M Anon, 119M Header, 2622M Other
=C2=A0 =C2=A0 =C2= =A018G Compressed, 25G Uncompressed, 1.33:1 Ratio
Swap: 144G Total, 11G = Used, 133G Free, 7% Inuse

If I leave this alon= e it will grow to consume all available swap space.=C2=A0 I'll try your= fix with the sysctl knob and see what happens...=C2=A0 I hope this is it, = I've been fighting this for a while now.

Thx,<= /div>
Scott


On Wed, Jul 19, 2023 at 11:34=E2=80=AFA= M Pete Wright <pete@nomadlogic.or= g> wrote:


On 7/19/23 07:49, Scott Gasch wrote:
> I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where,=
> after about 10 days of uptime, my swap begins to fill up.
>
<snip>
>
> At least they agree about it being 11G.=C2=A0 Is this kernel memory be= ing
> paged out to swap?=C2=A0 The machine has 128G of physical memory and i= sn't
> under very heavy load at the moment.
>

Are you running ZFS by any chance?=C2=A0 If so its possible it is trying to=
use as much memory as possible for the ARC.=C2=A0 I've seen this on a f= ew
systems which lots of memory.=C2=A0 One way to tell is to run "top&quo= t; and look
at the ARC stats:

last pid: 71322;=C2=A0 load averages:=C2=A0 1.02,=C2=A0 0.94,=C2=A0 0.87=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 up
8+18:38:34=C2=A0 11:31:26
376 processes: 1 running, 146 sleeping, 229 zombie
CPU:=C2=A0 0.6% user,=C2=A0 0.0% nice,=C2=A0 6.5% system,=C2=A0 0.0% interr= upt, 93.0% idle
Mem: 3599M Active, 18G Inact, 4132M Laundry, 4272M Wired, 892M Free
ARC: 1749M Total, 651M MFU, 239M MRU, 1864K Anon, 13M Header, 844M Other =C2=A0 =C2=A0 =C2=A0 216M Compressed, 758M Uncompressed, 3.52:1 Ratio



On a few of my larger memory systems I cap the ARC by setting this
sysctl knob (this is like 45G on my system):
vfs.zfs.arc.max=3D45000000000


-pete


--
Pete Wright
pete@nomadlogic.or= g
@nomadlogicLA
--000000000000d8dc6f0600de5111--