From nobody Wed May 11 00:49:49 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 82A9D1AC1F80 for ; Wed, 11 May 2022 00:50:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.gq1.yahoo.com (sonic309-22.consmr.mail.gq1.yahoo.com [98.137.65.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kybsj3bydz4slQ for ; Wed, 11 May 2022 00:50:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652230194; bh=wplqi3e0gcZ7BtbfygQiKnh7hTiSWnKvwzMPGWN3hjM=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ABgn/eFwKhaNBX2n2qM1VOZPWjBQTVwujN0HQ7fAPNkS6hOzlvAcReXA+ecfjQQRCDdod2YyEX5RmxEA7hpQFIT23ZFPIfB8YKt9jJIK/t5d9Sp9V756dKX0aiglB8IgokXtVIACuAsNnDLGCbxj4LcqQhAerV8ho76FmSD3JClVldJZoyyavDKj5AosU4XB7kyGgATXKbnY/Qt8/Uok7BgySSStVUb6sHmHJ7ZtdNVuzpziANVL4RhilbaJ4j31AlPnlMT7PIQCNB0bsmQvJ8k9gODbHhUMczR7X96bOS0mCP8spKe3U+1E97Sx/YC9h1CGuus/5EFuBDXyeyNuiA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652230194; bh=4xLjo3N8iAcpDUXcxyfi4FGVAWS5jINe0P2exEwS1Sh=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=N34Svhn5udGD9VkJ5GJozSo0gQQA0j54agJP8jnXATYSBE53D2Sh8cVyB1HlxnPEe4F2CIIlHVjy5iok+SG5o3FdMaJ/MczOQ7esR7FXHKN2JpQIk6dNlUHJ6e5LMeHmy1HV7sTZT976GBH8D0KyNiY2DgT3n/S0Ro0o7H+h3KV2zdvFlX8U9fSCf078KLbeDYet8cIs90UhICCBOKpId/8ELyrXerHd0diPAN7RtUaN0I9WBS8Rt0jzpapyVHjM3q5rlqhTKyQk2BQ8Uzx+9wowF8Y7p/a53KUdn37DMxGrKA2qd3OvhDGrsgJHAyEi5unvIWTEnetCaiQ/UOzojA== X-YMail-OSG: zvg3_uwVM1llEa5eeWCl6UVcF6AF8dFb2fyxlC.cQwDN00VS2k5Ru9JTViT5d6_ kmWfm5Cr2oPEbdtgrDJa1aO2odBio362pT4LNT48A0tN0rNfdwoOFvYbVOahDjhTM2EawbZQheHr N59kQ3c3hKf1sCew4umUHgRLCyh07Be2UcGUyKqGMnMbaNYJxtOKQ.rBcBqyy_fS4ZIK0jMq1fyr ULJnzAUn5ZeHXE4ZfSAASMZFfArGh3U1YFSvqTEK.A6ajnZ.DmlqMAiF89tx12JOnQegq8lfdQdc 2f059MFZTXgXEbfNtZc8O9RUGO0mMKI2Dlvncc_ES3Mfz6i7R_dA01rnxxvLldpNKleo4F_7ztAq xHDrmiCX0AK1SlscgA_TE4vSO4Qb6RdCcGF2LXthI1iL9Gzj8agUnhfWDyfkEmhtJFKh5JoZcrt9 EHfF8ix6UyB_ZtRd5On9qfwJWPwN8xq_w4ewu7MWiqvAHEPEBUE1RB78_Nvih7jIdFXOxsUT2Gwr Emgw2B5gTgPUK241F3CSjGAQ6.dAGRTdxbwyMLOK8Mwgrhoc4Zi7PvO9eofbDMY_kCh_H1n_hzsL QhK4FGbAhyX5UwuprdrVPfxo6vS8I52qVe.7AbIWq4QDXEiXxtSHIA1GYtF86mGhYX18EAdMOV47 t70R769S7_XJwjcRdUFTAv4QJuooX5oS75kV0WFgyw8pmVwNwRVKOOtPvsZAGW4t5gI8kuAFzAwA 8MIF2G0nRNFgXXZwpe85_KqItXFHg9eJGH5maVGX.j0ZTAsUHpVBa5NmDu.vVBdKtJoQPBATrEWl PnKbPDnEZ_lvtfKHMHHe1O4CBJivHzpuihhJ8tmwt37cAU.Mp5NiS8hBLfY0sbLQjaALzLlkBOl8 R5VHiQtfNXMsM_QBFHG3PdBRjB3MogtUa1xgWJFekxcmGxWuFBqD_tkBWlehIE6c9IBQgVT3qce2 zS4ML7SW_OHuwhUBpfQKahGLCWFHjdWRdCvwuCKcvUUbaQMCkqiSejgkvV1qC_Fo5lkhfIFzhZlZ kta.yhXnUK1JkZ2d6UORNJyLGHoZy2Qn17nLd8AJX4K5A_s9QGb4QhVtv1RLPKbe0na3s1QvnV02 e9kmqm9ZdDEIQTrb9p0y3fuuffGokAYwim3EfNWg6k4LAYH4TyzPL9p3Dd9sULTYMGcJ9gwVH3WA F6bteU4MpFDPQUCy4FWeeHpdOTZ3DClz8GqvXtHEc6Ujp25iugnnBNllLS6tFltLcsMfyuVj6Mt5 nEUWU20Q6trkTvr7_OQGi51lZAsA24sXoPcfq1GwD8B7o.0QXNwi0O_QOQWBaNJkHqFt7mngm1iZ 4iTMhWaUneCCttPg.MaC6WMiXUrNyzZyNID_Jkv9x9EgA95LcRdioYnHFdSOWbnHSgwWEPuFCTKp XjpCAem_2g6J9Op3DqIq1dfhxxd5R8ryiZx3ydBU.U2BDnIdnDUaeJ3UlhB3DDDyaIRaERnthEvg TIsm9ilz80YyJq2e8HtwNfG7Y5kTX9TONCZqjE8_tjmrjUYht9HVnu4sA76PCqmYryS6RLj.gXBk pWPvTtHHo_EqNOgJReAqeIwFjlzSCER3PlRtcjM1a0LVA2G4D2f.oKR_gM1EPhriru9ZquHEnXWY QGB3qqhFJyEKRT65wHCVcqnCBDdBWb2p979FyXl9aTsL7AGodYT.MRlrcQJNXdo1eRYi_tNkS4Ii gvDcjh9IlCtooUc4wyCVj.sLrO_J7V10LUJji14_pDC46xheh0bI_5nM0m41hyOlG9QQUCp2REpz byLs_cKyaZk_Kjjf9CHFWGY3OTcvqiCFygwAnVkXx0tX5e104v43C7l6OeXGowbH7dH_GYUkGDBj ITA1y_cx_KH3.ksjhOV833EUsndEIIBdJ_p1Mdo1Ry18sNp3j5XfeXB7QHTKesjeP1SsRUnKaZLS fFztSrgzpgYvOZSiljh648zH52ID6FxkqoLQYGKE3Vi0OAB_aYT73oATXHI.FLfyIJBKD91weDi0 X_n30SHyKna3rC13lQLR3EcT55MIlixZinFOLakqlR3E5HdUOkQxV_Au6eK.fqJKeP34hagGANXz xnAE2ymWQBoedM2zAsoezTub7uoHgQ3uT_xvMrxIvZYiuMNWAO1n2pY13pI1SMzcuIivN.M3x2cy y.oRRZd0IoRkfAK7k7LuwyR0f97yQugw6EmI_dw4r36qqpAqQrrOunml7k2_7Uix1qmZlH78rbYb jeIh6CqYATW9xfNcf.ZDke6qEK3Z8QXbjJ9Myfu3pKxaeFnW0OmceTXI63x.s2nx9WeXVf6ZV.u0 DHbxkgtezzJFk_Cs.7sql X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Wed, 11 May 2022 00:49:54 +0000 Received: by hermes--canary-production-gq1-55bffbc6f9-bklb9 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 9234404eed8d8a486ddfd38ff39546f1; Wed, 11 May 2022 00:49:52 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Chasing OOM Issues - good sysctl metrics to use? From: Mark Millard In-Reply-To: Date: Tue, 10 May 2022 17:49:49 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <83A713B9-A973-4C97-ACD6-830DF6A50B76.ref@yahoo.com> <83A713B9-A973-4C97-ACD6-830DF6A50B76@yahoo.com> <94B2E2FD-2371-4FEA-8E01-F37103F63CC0@yahoo.com> <0fcb5a4a-5517-e57b-2b69-4f3b3b10589a@nomadlogic.org> <464ED220-0DE4-4D2F-9DA2-AFD00D8D42B7@yahoo.com> <446d5913-a8c2-7dd0-860b-792fa9fe7c5b@nomadlogic.org> <33B740AA-A431-49CB-9F27-50B8C49734A2@yahoo.com> <3C5C183F-1471-4139-A53C-0B3815CFC25E@yahoo.com> <75C02C8C-6A5E-4E19-AC7D-B5DB704E8F16@transactionware.com> To: Jan Mikkelsen , Pete Wright X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Kybsj3bydz4slQ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="ABgn/eFw"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-0.02 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.37)[0.371]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(0.11)[0.107]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.148:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-May-10, at 11:49, Mark Millard wrote: > On 2022-May-10, at 08:47, Jan Mikkelsen = wrote: >=20 >> On 10 May 2022, at 10:01, Mark Millard wrote: >>>=20 >>> On 2022-Apr-29, at 13:57, Mark Millard wrote: >>>=20 >>>> On 2022-Apr-29, at 13:41, Pete Wright wrote: >>>>>=20 >>>>>> . . . >>>>>=20 >>>>> d'oh - went out for lunch and workstation locked up. i *knew* i = shouldn't have said anything lol. >>>>=20 >>>> Any interesting console messages ( or dmesg -a or /var/log/messages = )? >>>>=20 >>>=20 >>> I've been doing some testing of a patch by tijl at FreeBSD.org >>> and have reproduced both hang-ups (ZFS/ARC context) and kills >>> (UFS/noARC and ZFS/ARC) for "was killed: failed to reclaim >>> memory", both with and without the patch. This is with only a >>> tiny fraction of the swap partition(s) enabled being put to >>> use. So far, the testing was deliberately with >>> vm.pageout_oom_seq=3D12 (the default value). My testing has been >>> with main [so: 14]. >>>=20 >>> But I also learned how to avoid the hang-ups that I got --but >>> it costs making kills more likely/quicker, other things being >>> equal. >>>=20 >>> I discovered that the hang-ups that I got were from all the >>> processes that I interact with the system via ending up with >>> the process's kernel threads swapped out and were not being >>> swapped in. (including sshd, so no new ssh connections). In >>> some contexts I only had escaping into the kernel debugger >>> available, not even ^T would work. Other times ^T did work. >>>=20 >>> So, when I'm willing to risk kills in order to maintain >>> the ability to interact normally, I now use in >>> /etc/sysctl.conf : >>>=20 >>> vm.swap_enabled=3D0 >>=20 >> I have been looking at an OOM related issue. Ignoring the actual = leak, the problem leads to a process being killed because the system was = out of memory. This is fine. After that, however, the system console was = black with a single block cursor and the console keyboard was = unresponsive. Caps lock and num lock didn=E2=80=99t toggle their lights = when pressed. >>=20 >> Using an ssh session, the system looked fine. USB events for the = keyboard being disconnected and reconnected appeared but the keyboard = stayed unresponsive. >>=20 >> Setting vm.swap_enabled=3D0, as you did above, resolved this problem. = After the process was killed a perfectly normal console returned. >>=20 >> The interesting thing is that this test system is configured with no = swap space. >>=20 >> This is on 13.1-RC5. >>=20 >>> This disables swapping out of process kernel stacks. It >>> is just with that option removedfor gaining free RAM, there >>> fewer options tried before a kill is initiated. It is not a >>> loader-time tunable but is writable, thus the >>> /etc/sysctl.conf placement. >>=20 >> Is that really what it does? =46rom a quick look at the code in = vm/vm_swapout.c, it seems little more complex. >=20 > I was going by its description: >=20 > # sysctl -d vm.swap_enabled > vm.swap_enabled: Enable entire process swapout >=20 > Based on the below, it appears that the description > presumes vm.swap_idle_enabled=3D=3D0 (the default). In > my context vm.swap_idle_enabled=3D=3D0 . Looks like I > should also list: >=20 > vm.swap_idle_enabled=3D0 >=20 > in my /etc/sysctl.conf with a reminder comment that the > pair of =3D0's are required for avoiding the observed > hang-ups. >=20 >=20 > The analysis goes like . . . >=20 > I see in the code that vm.swap_enabled !=3D0 causes > VM_SWAP_NORMAL : >=20 > void > vm_swapout_run(void) > { >=20 > if (vm_swap_enabled) > vm_req_vmdaemon(VM_SWAP_NORMAL); > } >=20 > and that in turn leads to vm_daemon to: >=20 > if (swapout_flags !=3D 0) { > /* > * Drain the per-CPU page queue batches as a = deadlock > * avoidance measure. > */ > if ((swapout_flags & VM_SWAP_NORMAL) !=3D 0) > vm_page_pqbatch_drain(); > swapout_procs(swapout_flags); > } >=20 > Note: vm.swap_idle_enabled=3D=3D0 && vm.swap_enabled=3D=3D0 ends > up with swapout_flags=3D=3D0. vm.swap_idle. . . defaults seem > to be (in my context): >=20 > # sysctl -a | grep swap_idle > vm.swap_idle_threshold2: 10 > vm.swap_idle_threshold1: 2 > vm.swap_idle_enabled: 0 >=20 > For reference: >=20 > /* > * Idle process swapout -- run once per second when pagedaemons are > * reclaiming pages. > */ > void > vm_swapout_run_idle(void) > { > static long lsec; >=20 > if (!vm_swap_idle_enabled || time_second =3D=3D lsec) > return; > vm_req_vmdaemon(VM_SWAP_IDLE); > lsec =3D time_second; > } >=20 > [So vm.swap_idle_enabled=3D=3D0 avoids VM_SWAP_IDLE status.] >=20 > static void > vm_req_vmdaemon(int req) > { > static int lastrun =3D 0; >=20 > mtx_lock(&vm_daemon_mtx); > vm_pageout_req_swapout |=3D req; > if ((ticks > (lastrun + hz)) || (ticks < lastrun)) { > wakeup(&vm_daemon_needed); > lastrun =3D ticks; > } > mtx_unlock(&vm_daemon_mtx); > } >=20 > [So VM_SWAP_IDLE and VM_SWAP_NORMAL are independent bits > in vm_pageout_req_swapout.] >=20 > vm_deamon does: >=20 > mtx_lock(&vm_daemon_mtx); > msleep(&vm_daemon_needed, &vm_daemon_mtx, PPAUSE, = "psleep", > vm_daemon_timeout); > swapout_flags =3D vm_pageout_req_swapout; > vm_pageout_req_swapout =3D 0; > mtx_unlock(&vm_daemon_mtx); >=20 > So vm_pageout_req_swapout is regenerated after thata > each time. >=20 > I'll not show the code for vm.swap_idle_enabled!=3D0 . >=20 Well, with continued experiments I got an example of a hangup for which looking via the db> prompt did not show any swapping out of process kernel stacks ( vm.swap_enabled=3D0 was the context, so expected ). The environment was ZFS (so with ARC). But this was testing with vm.pageout_oom_seq=3D120 instead of the default vm.pageout_oom_seq=3D12 . It may be that let sit long enough things would have unhung (external perspective). It is part of what I'm experimenting with so we will see. =3D=3D=3D Mark Millard marklmi at yahoo.com