From nobody Tue May 10 08:01:52 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 22C8A1AD801D for ; Tue, 10 May 2022 08:02:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-23.consmr.mail.gq1.yahoo.com (sonic304-23.consmr.mail.gq1.yahoo.com [98.137.68.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ky9Vm07x4z4Qqn for ; Tue, 10 May 2022 08:02:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652169719; bh=y6FauHuBTXpRD5lVsu35PiXIGHmwod6NCGC8lQDPs6k=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=lsT7ZgK9efvbZt0L67X2C691Rv6aOvYoKBRSt/OsBLxqnl/k9d+uEKSLz2cz51YPfDTLMyyEA8ahe7EBsssbMtsdFdVdKacwtUmIdf9OWKsEdS/HbvSCVVRs0ONt87feXAy27HoSGZeShXFl3H4t5jbCnA4SSJccZxQlrMEn/ChB7MKDnosVz4cBOu2216RyccSq1/vpfLcTSSZEmKeVLTe87DSObnJ0i2n+E8RRJCutD6HTJEVQi45iF7nW/PyuGZtiIGAJQegYGNqdQZkVg5R5an8WS9nbJq9mqUB/K16L3r5nUFq/M8j+5ejvoFof8PV3JGVqCo6yKsBqR0tYlA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1652169719; bh=BBwh2ufIgLkz3zU/DS/UW1PlNP78auj8E97u3hUiHIi=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=uRC9RfbtY0gvtLv2iXdC8UMPlVeU5WiG3eATsGft7t1fghhiFb53Ztqo1F//DcCxnW7JAlYByl+aCau4HvJWHWE47ksHUfEX1dmOP3bItuTljfQoVQqNIOxpiody8/JC2ZrsJXgV1bIPC0EE4gD4WnSkO2NjP6llqSKwKvCwTehfdk2DFPEpDxg6mOOTwRXPV7LGwWX3S+21UcCHBH5Pr5+VX0YIiS0umCAvdOV8Vg/HtohOEoZrQc1FT7gqzUuwUbTS5H3cnm1tMVO1bo0ISzQyhbWz8Zx5j2VPf1F4qHPLsUSZP2N1z1eNolsovyh7FMRu4KF8Oa8ZAAQSe2z6mg== X-YMail-OSG: rifv4NIVM1kdjmQz1muJyGVF4pc_v9XG.mMJLxzXjEWI3CHi2MhUwVYnugp8vP8 5vF4qoEMBGhVqB9AqaVjaqbxVhKskUNpyjRmaixODozzEeMQPOvTLI9wXjF1ytp5IqKaKGxQohus X44usTnQHRrFtejA3utoSh_xVpbLU0vHQDUu_XwGYLpH02HKdsWfPXj4Agrya5XCLgi0ItVntv8k Jr9YY5Q5QORQ_PakFIKktDuBlWb_K80lPgZM4dWzOLh_js_GY88pbxQhFHAUxOCNFnheMHHNUnGq 4rGpRx2zD4inVBjiYUvzM5Yojb_K.QwW0swWElEwLPl7MjJ.4gms69viLiMDXaCkNi4pk3k.FuL6 0.42igUxxzxBMFPD6ff.o91Cr7.ElOu38MibEGDop.XPUvHBXo1z7upoXWp2dwHXimkQ1nzIcwSS LIGrmBZGP4JI4A3hDR39fEdypfA3_XrBEChhu1hu432BE6CnPAvK7juJXrfFz71d0JMCU9CAz3I4 CxLB7bfrgB9Arg7hzUO0LI9tc8.XrWZJoH9R9WtzjoB.ySk6uagIZbIGo204NOkAwFQC_ZG70wwK LezSS7DjMu92JR4ONHyt_Ir.kywrg6cq_cLsnMkZ.rmQZ4NaZuN4OJPpg7aXwLBV2bEpWHVkpjCX I728oOLgIE2x7aoy4PKELAnXOWFFe_F9BSrsc1Sw9VH_.ooPtTsDa.aem0T.DDNplUtkkPgsOOAX Pfv_1PtMIvOCNTkdfdzX1PyDDDN5lS8vxZjX7F44FPHSlcrHZUCe9bTh_EyGWblVM1nWRzTpt41j 8G5mwB3xo4WCBCqyGd7zewbcpQvceiNSdEFTK3UNImjUMT5TZl_0xTOmdPmVAcTgkMZRq8fanJff C8_zKXB8S22l6nxrsOHuSD18vu2xCdp4H43ve7lXVyrwIamHCILyQ4Kqsr.KDW7szf6OhGxWBzu. uGxgdep05SGboUahLqdcbooCpG4BOt58ntOXDCA8brnYDZfV6Q1pn8dJRnq1F_8NwKUumhYazL0f 2bpG.fIy4KgbTlfcnjGA9MYm9s2ZpEbVkhgUJHrESGua76jcvFTrbCRjmHvxB_sU_zX1ftPAbyJJ Ft0W0a1TqHBEgGUtvC8tqMWMdfhU7FyLinl5sDy5XXzGc5ckp3pLsvBrIljkOTjQMJlPoTRQPimL cK7c0HJ7Lxw4h.yctQfXZ2FNAB9ATC50ecuzLe2MjxBROmJ.r7zE.ft9dhBiBcS1qflgAvZ87Wsc B4tJX9nM98IBI__buZGTJ87lHYoaDd1xIhlZHdLSvxS_hpM_xOZIBKx0.O2uKXM6ANkIPNQqh936 bBALD1obBK66RuP8QIAMDwKX68Rjk5weDU_Yw2vR8kOhAx.mibn.BOEKKAQ6ij0b46PB0g_YyFka 6mwdDUqVATTc9br3avfGDTkTCyuYggMIMtQRgBoWpUIlAd97k6w47ayiRPY9ov.WsluBIL6uRf3C hAGHH1jKYlIGmVsPY538Lh39SHL3faEDFTqi5dBB52yU0YaRyKhGvGxR3sAqW3EagmBRgOQ0IErh edDTXzAjdE_Ywkb5dI_mDZ7y__FF05Wfc.ghDZTQTgc5sX5WasmdZqvudk3620a0swUJrfdKd78I SY4cUJ.3aQ2DuVi4SFhESc2yDpq8xwWT_nxSx7PObPlBV3SpRVfbHMXqD.xEjfABI_szDtmhgrUC yaIDPDkJEUUENxxdPCpVExYcOunpNo6btqcgH3znxz9J.xZ05rM1KDc8u4TcIQlqKYfFul8eKoZ7 hoUYPY5CR6SqxiO218qWRkFkoIFzWXWWGyDN2aKBk_UFEpC7MWWD52gsH_7_EFT1XGo582cWqJJT W66_sYwdr.DnTElR2bi_5UtwR1Z.s.dbwAoH_58T0WXcgjztKLa86ftCYdkKcRPVq3vbqub4dQbL 0tABMiuTJUAqd6VUSkiJOfTg1I3viEo.LWqaUpvw7MEolYTXOrO1NjBxt.bTtQFVgO0qnO5EDcPG TsJqMDRMId3VEj38C9EIM9c1Oz7R5.MS90dI0k3ITeZUxsjDcuEhvl2V6m_TPyJvwz585Ht3yW9p q3QHuGtZlL45VqYE9.oYgK4OlVwoXLqm23.6zAdLk9LimP6SlJFrtnwdO.rp969teyepyyLk9NHm _LpgapdhKWM4dmMRQKHpBj67jmLBAv0yL010Lia6aD7sqSFOODpMGQ2JcjYz5HWf80Hyp2LXj1wH u7Kj5m4fQVLCo.Z_Tf.Jso2UCowwK3WV2dOAzINeeBfEwaGAx4c28iGaJFEV6MkqbLJ1Xj7GMG8P Yia2VPE1C X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Tue, 10 May 2022 08:01:59 +0000 Received: by hermes--canary-production-ne1-8676f67b88-hk4wg (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 56389975409d266b5648baef06b84a82; Tue, 10 May 2022 08:01:56 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Chasing OOM Issues - good sysctl metrics to use? From: Mark Millard In-Reply-To: <33B740AA-A431-49CB-9F27-50B8C49734A2@yahoo.com> Date: Tue, 10 May 2022 01:01:52 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: <3C5C183F-1471-4139-A53C-0B3815CFC25E@yahoo.com> References: <83A713B9-A973-4C97-ACD6-830DF6A50B76.ref@yahoo.com> <83A713B9-A973-4C97-ACD6-830DF6A50B76@yahoo.com> <94B2E2FD-2371-4FEA-8E01-F37103F63CC0@yahoo.com> <0fcb5a4a-5517-e57b-2b69-4f3b3b10589a@nomadlogic.org> <464ED220-0DE4-4D2F-9DA2-AFD00D8D42B7@yahoo.com> <446d5913-a8c2-7dd0-860b-792fa9fe7c5b@nomadlogic.org> <33B740AA-A431-49CB-9F27-50B8C49734A2@yahoo.com> To: Pete Wright X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Ky9Vm07x4z4Qqn X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=lsT7ZgK9; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-2.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.204:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Apr-29, at 13:57, Mark Millard wrote: > On 2022-Apr-29, at 13:41, Pete Wright wrote: >>=20 >>> . . . >>=20 >> d'oh - went out for lunch and workstation locked up. i *knew* i = shouldn't have said anything lol. >=20 > Any interesting console messages ( or dmesg -a or /var/log/messages )? >=20 I've been doing some testing of a patch by tijl at FreeBSD.org and have reproduced both hang-ups (ZFS/ARC context) and kills (UFS/noARC and ZFS/ARC) for "was killed: failed to reclaim memory", both with and without the patch. This is with only a tiny fraction of the swap partition(s) enabled being put to use. So far, the testing was deliberately with vm.pageout_oom_seq=3D12 (the default value). My testing has been with main [so: 14]. But I also learned how to avoid the hang-ups that I got --but it costs making kills more likely/quicker, other things being equal. I discovered that the hang-ups that I got were from all the processes that I interact with the system via ending up with the process's kernel threads swapped out and were not being swapped in. (including sshd, so no new ssh connections). In some contexts I only had escaping into the kernel debugger available, not even ^T would work. Other times ^T did work. So, when I'm willing to risk kills in order to maintain the ability to interact normally, I now use in /etc/sysctl.conf : vm.swap_enabled=3D0 This disables swapping out of process kernel stacks. It is just with that option removedfor gaining free RAM, there fewer options tried before a kill is initiated. It is not a loader-time tunable but is writable, thus the /etc/sysctl.conf placement. Note that I get kills both for vm.swap_enabled=3D0 and for vm.swap_enabled=3D1 . It is just what looks like a hangup that I'm trying to control via using =3D0 . For now, I view my use as experimental. It might require adjusting my vm.pageout_oom_seq=3D120 usage. I've yet to use protect to also prevent kills of processes needed for the interactions ( see: man 1 protect ). Most likely I'd try to protect enough to allow the console interactions to avoid being killed. For reference . . . The type of testing is to use the likes of: # stress -m 2 --vm-bytes ????M --vm-keep and part of the time with grep activity also running, such as: # grep -r nfreed /usr/*-src/sys/ | more for specific values where the * is. (I have 13_0R , 13_1R , 13S , and main .) Varying the value leads to reading new material instead of referencing buffered/cached material from the prior grep(s). The ???? is roughly set up so that the system ends up about where its initial Free RAM is used up, so near (above or below) where some sustained paging starts. I explore figures that make the system land in this state. I do not have a use-exactly-this computed figure technique. But I run into the problems fairly easily/quickly so far. As stress itself uses some memory, the ???? need not be strictly based on exactly 1/2 of the initial Free RAM value --but that figure suggests were I explore around. The kills sometimes are not during the grep but somewhat after. Sometimes, after grep is done, stopping stress and starting it again leads to a fairly quick kill. The system used for the testing is an aarch64 MACCHIATObin Double Shot (4 Cortex-A72s) with 16 GiBytes of RAM. I can boot either its ZFS media or its UFS media. (The other OS media is normally ignored by the system configuration.) =3D=3D=3D Mark Millard marklmi at yahoo.com