From nobody Fri Apr 29 18:38:13 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 531A71995C0D for ; Fri, 29 Apr 2022 18:38:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kqh8127pKz4s8X for ; Fri, 29 Apr 2022 18:38:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651257497; bh=uV5bxg0Ze4bRMxG+SKLIdjMc1bbIt5NSBKoCsQuhYq4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=WGEgHAjxTcwDbVeHgsqOuQAPsmmGDl9Fw2fHb5qMgbKgQVsx1VKwoFD75eZXOyhUJLI+1MfW/Otqq5surArbr1rXM5rPKVzMsRp46KbO+9LmUcn3qWJE01zpCebc4859LXzjZ5xYhTQKuEhSk2fyaiKa+hU2+dLhHP2QqbKUg8i981YxeFeu0s6spKKXBtTspIQVGKsAcfAfNRRIrwZVQnY5n6rzHSEywbHHSt5ufgI6yRT6g89lgcRJf9ShBz3R7bptlkjgmw+HKcdhtSdDtMypjlywCpI+kfxeUpxmqto3S7ip7EuApFA13Y7gGy8pDWny6qHD9Sy8Ia9dLa0hYA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651257497; bh=lkE7QsBsalSQ0g8JXwYUCpeTZNHstbjrOwvX7NjA1/q=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=kkPwOSCm8PrVCYSyFGfiiiu383Fzulpd4/1pPuXgamITeuOKxcs5V9S80lciAErcza2bEIozSX75tZsDJ+z3SWKISGDwo/Tdtsgfa08E1f5BU7PLKELevwyGlAhZtBnP6NNttBhikAFFCgFIcIjuedDBGRMD6sXV0EzaBJv5WG1i0QKzamcT6t4DrYugJQxJRAycIkV3hFd2/WvFUiSH0TZwNzKEnuUUufgd0cGx5WOLfyHjj0IaD3uu6UYJNvJ11cqNoFNYggcUJZCp4oCh4fmWuNRpoKvyVpklLf6OhKXu5e21Cy47z6/Ji5RThKFbWi/2201F/77Fn6jDLNeP+g== X-YMail-OSG: TTP8jqAVM1mctnsNKWZwSWWskVpaeRhNteiUOdexp7vxp.HCZ3yvY4WPjqxxG6M t52Jia9uCR.OoNFMUeNN9iTDZWBD0iH0RYA2vsyoTjpoOtZRjdTsfjp.HBB8nJGRE.wI7.XyIgyH wLIwJubcOhczqbAQWizIwaQY3Q3vNVfDvexl5mEXbD7dNGC6EkFqo3TUbpWDZAccjNbDGtzmqXIj jGvfY6h4gG6vXnYf7iBb1RPMh3Tg0LJqjHbB5Zw_VU.lMTxPHIiZ1MXIvbsmPa8_i1JcNO6Cp79I lUz3RdKmU4jFbyOQ3ydATmlA1dhn7RBcnWswKhAAQfqYEz6WostyJ2q6UcfkbwJkMZHqItuqJWQc sWzU7WY4wxiIeyeqK2pLli7slLiU5R1ycjtD7_zQa2FIR44kU_ZmBtXALrJ5WHDxyfB4Z3iynLtC oll_js0L.46Qbd96KDe5xX3Zpfj0yD5h9CH6goh7QYt0y9fXKXGeZShrtWPk3iOCEiVtfer.9ovh YCAlvPJSLkcNg8945yxFVuHfs8BiBDQWolWxHjOcSf6aMhcOopRMh3ElIo_A2yVqiS8DQxI4nTah qfCOljQ76WhEbL0APD5jmAY5Zo6gpJAgy8NGUHbrTcz2BZXo9QO2bITs7QCY1jcZ3r9.sDwTxBtL aobaGB8jTgNUvnsUnQ7iA_kNuXCHFYrWIS1kqfAGas_E4hBqhHeSZ5lgzh4ZCjGjJlUP0YsHU9Vr eOfygwyU2J9jXoZQ.i4JxYEXhmyBZalFFzY4Yf4ws1bWSGZKY5PLruEMSwPBnnzaTNihNs2H0sls Ar7XBbakztO0unHs_NgCn.TusbuS52CPpnzhgATbpqNrOAuV6lPQO6Fn1IE1uN2qzae1Wm3YFuiK v9yOLyXBFi3xXHlNQXwS8MjkqRRCV_4SNcQQK7hzD_kcmD6fUEvJP0SZLGDA1XeO8hIBUgRmSY5F 7i.CHYP156ov2GTwaPt9nODLkfHWL5LPl5ZHqodKtiFmKgbD4QFFABL0CXYAnb0kdc_2iYIcjsbe sgUERF56NUOY7r6g2rW5rF98OOxbCB2c1_qKA9TmRZ6_P8voE2u0LFgD3wKghL1_ZK.pKnBIMTwH RFxXwiq6UO8g3h8EC27M1zwkHRkYhbWrydy0hclUiabZ8NRkCMTSFq_lD7KYIy75DVrN5tkHUrdi cqJwNt8MrdiIwIdf2Z9CMlEWX4L1zAOwco_HPz6SZ42Qrm8bPjLJnB.nR6BMOaR2o8nHbrFah5A. YcCd1sPF8DUSJ3NIcatSLZmjoAUkGjeDshpkVwux0F37vh26lBAev662kgx.111iGV_fSoqMKQcY TLln7dirz3qmqZV69BuXaQv2qftn2XhP0gSiK8w5mnS2m_HMmTkwFUM2LUpOtFD47yzaojbV62V4 AhovU_OZHE101vzpOxPeQ58zhOdEMwrREI2nazXF2WtZRsmLehc9siVkmp4wxHamUomZA2wQFvmM 1xu5ig7ewn9MtwDxMfJXx1sKBohkgTLCSWE0DuuOR7go5B9NLlHHrFgz2UNanH7vWvwc6jI4fSSg 4Jz.vOD3T.AgtoUp1HnBPS1H9J0qUTUY9nCtUEcjgNANzbkFCppznbT21oeqq8Hq4eL8l9WiBIOl Oo.8gWexs._vCn5iKO.bcoVw4N93h32SGdS_0TiZR.Bd18RQqvAvqqIYwEu675GvniQabELSAeZT 4YzDt0f8xGoY_zmt5RZE4nW1rHMdlczb474QN9KPtoay1YYPKhTJwgFjqvkz7m1TKXRtozQYOzjU f12JwTo1G.ioA9ARgM6brypNHGWKZVHgBtZLwKVrQJoZ2CPD1FSPAE0iFILDuSvMl0fJCQV.5Rdt OnMvQhj6iPX_Q33Ty1m7OT8h8NGHpdgU7ddZgDf_6FM8bWslwGhrBNlAGQfluXLo104S4HKw_vwe .IXdir2N8wtzQww4d.ycwkWQhVMQzIF4FIUJ08EvLETcNaoLxbuyvQnURBrnIpKuLS7c4L6Xnfyx h55jDP75.NnBvkeuJkl98xVm4aeY8kiSX6TnI0uDLG12y7NXQZu8GphIeirt7Q7oHw.2cFdTVxAa a2D.kXEicKtJofpZJXncP3VzZuztjFRLyt82JA632ebYOqM8V_Jh9u.0K8PobGr77dJgIlRjXLcI sFn4Z3ARTSEekKgYhVdViU2hzJAxP6zgbyyBnovv74vcIsZK9G.y4KnuZth7_EsthOwYa2L3_Of6 dGvUTC4h8bVVT660xySJN_V4OCii2HFTVS2T4IasPjd1l7l09f5yeC_Ye2T55jGXWl0qxobAD8Gx tr_2Ea75FO6tC X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Fri, 29 Apr 2022 18:38:17 +0000 Received: by hermes--canary-production-bf1-5f4c6455f8-qwrqm (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8bbf46de4c52f96172fec0e517f34c52; Fri, 29 Apr 2022 18:38:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Chasing OOM Issues - good sysctl metrics to use? From: Mark Millard In-Reply-To: Date: Fri, 29 Apr 2022 11:38:13 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: <464ED220-0DE4-4D2F-9DA2-AFD00D8D42B7@yahoo.com> References: <83A713B9-A973-4C97-ACD6-830DF6A50B76.ref@yahoo.com> <83A713B9-A973-4C97-ACD6-830DF6A50B76@yahoo.com> <94B2E2FD-2371-4FEA-8E01-F37103F63CC0@yahoo.com> <0fcb5a4a-5517-e57b-2b69-4f3b3b10589a@nomadlogic.org> To: Pete Wright X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Kqh8127pKz4s8X X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=WGEgHAjx; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-0.99 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.51)[0.510]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Apr-29, at 11:08, Pete Wright wrote: > On 4/23/22 19:20, Pete Wright wrote: >>=20 >>> The developers handbook has a section debugging deadlocks that he >>> referenced in a response to another report (on freebsd-hackers). >>>=20 >>> = https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/#kerneld= ebug-deadlocks=20 >>=20 >> d'oh - thanks for the correction! >>=20 >> -pete >>=20 >>=20 >=20 > hello, i just wanted to provide an update on this issue. so the good = news is that by removing the file backed swap the deadlocks have indeed = gone away! thanks for sorting me out on that front Mark! Glad it helped. > i still am seeing a memory leak with either firefox or chrome (maybe = both where they create a voltron of memory leaks?). this morning = firefox and chrome had been killed when i first logged in. fortunately = the system has remained responsive for several hours which was not the = case previously. >=20 > when looking at my metrics i see vm.domain.0.stats.inactive take a = nose dive from around 9GB to 0 over the course of 1min. the timing = seems to align with around the time when firefox crashed, and is = proceeded by a large spike in vm.domain.0.stats.active from ~1GB to 7GB = 40mins before the apps crashed. after the binaries were killed memory = metrics seem to have recovered (laundry size grew, and inactive size = grew by several gigs for example). Since the form of kill here is tied to sustained low free memory ("failed to reclaim memory"), you might want to report the vm.domain.0.stats.free_count figures from various time frames as well: vm.domain.0.stats.free_count: Free pages (It seems you are converting pages to byte counts in your report, the units I'm not really worried about so long as they are obvious.) There are also figures possibly tied to the handling of the kill activity but some being more like thresholds than usage figures, such as: vm.domain.0.stats.free_severe: Severe free pages vm.domain.0.stats.free_min: Minimum free pages vm.domain.0.stats.free_reserved: Reserved free pages vm.domain.0.stats.free_target: Target free pages vm.domain.0.stats.inactive_target: Target inactive pages Also, what value were you using for: vm.pageout_oom_seq ? > maybe i'll have to gather data and post it online for anyone who would = be interested in seeing this in graph form. although, frankly i feel = like it's a browser problem which i can work around by running them in = jails with resource limits in place via rctl. =3D=3D=3D Mark Millard marklmi at yahoo.com