From nobody Thu Jan 27 19:31:17 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A9D4319821F3 for ; Thu, 27 Jan 2022 19:31:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jl9gl46HMz4mQ7 for ; Thu, 27 Jan 2022 19:31:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643311884; bh=yhZgD+NpPfSMfC5l8LvISuGzmSJ5syvtQqrQbGfbyqk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=BB8nVxWtKPMTwWMlh43IzVgOtVXGP0ayerJrwPpkYXdro0CLhl8iH3LyGayAV1F2LV6mElet1jmEPTAkXO9tzKkaFSetpGP9QH1vcMaV7p3Moyr10eLpSLPztQqUvz5R1BaHHyVYy9ek0t3JjwJic2Ru5l/EmfcDnm4sBYo8h6rIg7hNttTgjjPn2iwW4Iy5JXhEkb9YmT4AyiwdWwmbN/TAIVpyzyPEzJQ4e+uUuQopWEIadGP+mT8so1SvGihsNWylS5bmyYm3d9uxTJUAfAX4wTNbAzTysRMKN5gt0RKpmMakQZkNJ9vjoc9KDf/KayuJrBKrbDUfjg9Q9X7BgA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643311884; bh=f1fXoKJSTMNny+wMItKAdSKVU6fWe0Rt4KBhpvasN7u=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=DNsMFfx6f8Q/0TSr77a2Cq+hllX/9nfuNH1ZuKB4T5KerwqEPEqqHTedjDteXIe5s6lFrhBTYtJQ3nT9tZk5FOUV7Ikc+PFNNv2LLAZ9Jz2wiViSnJlz+FAmJ/TxLBcRf5YmBXyzDSVWjWR+jc75EJVlXzmh5Xfsh3mlQL4lydCaKrn2Ua7QCGl3tugFPY19YMJt/l62/jP7AqysD30rcbPJt5nlyHfh30QfxMj9bkXFu8kYISIZL82qiaiaok1BS4R403sN7qBGyLahQ+43h/GGltePLQ+V/S7UTemWUrtStCiZuD6WE2cV71xeamfDEYGll0Eu/9trRRSh7jgmeg== X-YMail-OSG: o.z_dHMVM1mYO7jUWSS_zAzLnPZRemgQ8M7OB28dVjSA8kq6zdZ2_OK3.LtsSgI RU8g0ynnG8e0pfBklbcU31kzZxkdZ7l_Ea11I5BvGDiMbonXgDy865LBHKsUCHSGRjnTB56.soWA 66HXT2tPhZze5e103fWKyzlZyMJ50.L.hsh7ZkERkKR.2nzAvT.ZBwP2guIOoI7GoCXrDrMu1pHJ fXbyACqnSY115xA8k.de3OhVVRp3RTow4Mlz3jjx6edzxLskndWi.Zy.hPDWqSRdMlstq2J1.Ztp cIukJW4lLR6hAccbSKtMh31gQWh4KIJkxlRjbuohMI6g.Md0XUExdV1aCTlL3HEtXeSQMqZ62jrK XPWIK_r93a0BM_9rXzcqEZSsPpLIxcwlavj3CjiEYWKMJhxhXQTracCnIbl7lEcY.YBo7xi0FJCM PR1K3ct5KHWxJXx1qM6khFAFON1YZ9lRi54GbpHhhdu21ogd41gDfdHG.pwhyJJqGnmybSct8j4o 5a4g7QL5kdT.xJgKoDg.T9sCwIa3FylETAHku4CEkPPrr5bb6Z_s0Jhkys2rq1hlEICqsAqk1P9M 5dcJj0RpceMJlHC2NVF.YWOmk89i1iFjqkYN4et6Ouv0J1eveAU5cQLCwbAf2aiaFESn77QPQgNF bRORcYzXxOCF8c4HKaB_NZP6Dr9bGHx8yj.oHXKjlS3XrYt_IAJ7ED8V8XbhM.aHU7cq2IwzZ6GH HnlhVS5IpcJ4JbuCmELPChbStqXHLFF9QaqGPynDua47KYdK6e8O5ogv_xD3bMFWJe0dLbiMcmCG u9h5d4f0vIVnH7k_9M8twb8CgYJOUZ6RGxYT0s.hG43b3FgMK7ahmTfbDxbT7qFxMCtJb.8lF7Ne jyaDNwJZvypkBn1zQc64Lqs1NqM1Lu0FnYbtgqX1mW5UGSWZkNWFqjuquDuYY8mv_TD9Hm9MxbMC JjEi5KV6Zv5U7byoDU99Oe66JPdyrHOze17.6Lct6CxOcsZ64lFR4h9xJaEA1QAcjqzPRpjvtINw ygsUuQUdp9rjvEyFR.bEZRj1pXfnoFo5yFASBZaIdq0v9OReUyoPcygSyIxGgmhZBkPIYmcqIL5t .Y2Vc3oyDm5ARYUyNjzLR89UTRyjdNs.npvvNIInQ0mig03.Q6pww3voDP6X2bGb_tMb0GZWBV0p v4SiKuUGlrxkMEiSgaJGKrrqaKJkk.hmiypIvsFsNT4ySDkCm10ko0J5IwbPr3xBjlK.tP7wb4Xs OVr29bwGAo8gwEJoEGY3L6WLEyjHR3Ic1cLlklsG15N2oAzn_4Fr9aptIUMjAZyyXbw3jbbxpkxg xF3L60i2sKpMgwZvnt0EuZtdDjXLNzJynhAI_u.29lcvMrAPjlCaobXB9UEHsVhke1obByQm7LQd 6YlZDt5Hb3gATP5nCcZtNbDpPsACe1pAYatCRTo0NlXWhy0.XutZWKOlX7XN5kjgOIjvJB_44I8U sBZcCzAFRUfLCafckteQi.g6dT_HDc87TybNwbmE0TGdwDEZ.zYW.eCm91.6wNZIXFb6GPIj8aB2 RxrqKQmr6ZrPvKzidPb5G5UFhbPOyb5dw59HA7oGcYvpHVnnB0JpHbDSyovMN2H95ReDddaFFwap Fj2dGCDNU49uJHsbXBKtyVOk1YvGrQMLmybjGVFhyqRR7UW0hqvpdeAUEIqLCg9IOjNE09JPR74B lukcNALU9SH7jxrmQ4y8.cF_S4ydzR9StQ3N.0xsPSBF8o6oDMMO7b1NIJyNWIUjImZGpsnRAOnx x7uPMeNaZwaE.PhZiNNqH8krFIBu4.yxIleRTZ1KySE.F.4DEW5lBBH7xn9j3VdfD4LDqMj5zJ8f rVEfRGrg58e4Oe0CpJ4gSOyZBbbXjyFBXehkwbl.N1EGbsHfz__BJbZvKpeB1fs31.gw3XinlzoG c5cf2lpCz36vS79WM7zvZtYbC5qhkL0T4UTnpmYE9mTmdoVe3f8ayZAIp7HkTUVm_fUSt_CQdIuF 8Vc8IWhs4avwaCobOT.RwFLLkSxtEkyjUqHorvgIeS5.YntUlV9qGJ_byKK31286GOAO4go206jJ _IKmID7EBqNrCD0W6BKgugzwc9D54j9DJgjy7K2HKJhsc2V6magCc7XhGOCDTBkdhp85SB0jz5uC rwuwc79vc1HBm_TQwyZxvftEtB08DTO1UMq8IHDxtUXsPmIt5vwfvhGt5wf8HJXB0ZAOSCdJLRvt grswUMCD73ldX1Cx8If5ZhdfadKmp.mk.hVaFDDi9jrtHTH6AL4IYdMC6WKTmEfdvggALTzduCxK z0.KM0dygrDZs28vnUnfYsOE- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Thu, 27 Jan 2022 19:31:24 +0000 Received: by kubenode520.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 6674841593c6832bc88b93800907ec33; Thu, 27 Jan 2022 19:31:18 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current From: Mark Millard In-Reply-To: <20220127164512.GA51200@www.zefox.net> Date: Thu, 27 Jan 2022 11:31:17 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20220127164512.GA51200@www.zefox.net> To: bob prohaska , Mark Johnston X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Jl9gl46HMz4mQ7 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=BB8nVxWt; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.31:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jan-27, at 08:45, bob prohaska wrote: > Attempts to compile devel/llvm13 on a Pi4 running -current (updated > on 20220126) with 8 GB of RAM and 8 GB of swap has failed on two = occasions using=20 > make -DBATCH > make.log &=20 > in /usr/ports/devel/llvm13 using the system compiler. The system is > self-hosted.=20 >=20 > The first failure reported clang error 139, but the second > was different, reporting only: > FAILED: = tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/check-expressi= on.cpp.o > along with a console report of > +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1258432, size: = 4096 > +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 627221, size: = 8192 > +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 240419, size: = 4096 > +swap_pager: out of swap space In recent builds, such as yours, the above "out of swap" is a misnomer but is very interesting for what it is actually about. Mark Johnston later wrote on 2022-Jan-15 about his "git: 4a864f624a70 - main - vm_pageout: Print a more accurate message to the console before an OOM kill" that produced the above report of "out of swap space": QUOTE Hmm, those cases should likely be changed from "out of swap space" to "failed to allocate swap metadata" or something like that. END QUOTE Your context proves the metadata problem really happens, so the messaging should be fixed to not be misleading. In my builds I've code that is more explicit: diff --git a/sys/vm/swap_pager.c b/sys/vm/swap_pager.c index 01cf9233329f..280621ca51be 100644 --- a/sys/vm/swap_pager.c +++ b/sys/vm/swap_pager.c @@ -2091,6 +2091,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) 0, 1)) printf("swap blk zone exhausted, = " "increase kern.maxswzone\n"); + printf("swp_pager_meta_build: swap blk = uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxb", 10); } else @@ -2121,6 +2122,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) 0, 1)) printf("swap pctrie zone = exhausted, " "increase kern.maxswzone\n"); + printf("swp_pager_meta_build: swap = pctrie uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxp", 10); } else The "metadata" is the "swap blk uma zone" and "swap pctrie uma zone". Unfortuantely, which got the failure is not still indicated in the standard builds. > +swp_pager_getswapspace(12): failed > +pid 61012 (c++), jid 0, uid 0, was killed: failed to reclaim memory Abssent being able to swap, it tries to reclaim --and that too failed. That finally leads to the kills. > Swap use peaked a little over 50%. So at around 50% "swap blk uma zone" and/or "swap pctrie uma zone" had problems, probably fragmentation related problems. > After the first failure a restart > of make using MAKE_JOBS_UNSAFE=3Dyes ran to completion with one = thread. >=20 > A copy of the build log, logging script and other notes is at > http://www.zefox.net/~fbsd/rpi4/20220127/ >=20 > Clang error 139 has been seen several times during make buildworld on = a Pi3 running > stable/13 with 2 GB of swap as well. Perhaps the two failures are = related. The Pi3=20 > failures didn't report out of swap, all were clang error 139 with = "failed to reclaim=20 > memory". Even with only 1 thread (j1) the failure reproduced. >=20 Note in your report above: obj.FortranEvaluate.dir If you use the options to disable building flang (a.k.a., the Fortran compiler build), your builds on the RPi4B will likely work in the current configuration. But it looks like you have identified a test context for the "swap blk uma zone" and "swap pctrie uma zone" handling. =3D=3D=3D Mark Millard marklmi at yahoo.com