From nobody Sat Nov 13 11:40:06 2021 X-Original-To: arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5BF06185F710 for ; Sat, 13 Nov 2021 11:40:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-23.consmr.mail.gq1.yahoo.com (sonic311-23.consmr.mail.gq1.yahoo.com [98.137.65.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hrtmc2LpJz3m4W for ; Sat, 13 Nov 2021 11:40:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1636803609; bh=k8Bkw9bJpQGpwQ9/O4FqVENpmpwJnbd8/tpTrg/ocM8=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=ZEvWZaO8csTl4V+lZ/ZiNrXHsqumBe8CDz/h6UMj1kxe894fw4gZqhfO5OmP++Wuh1JuoLCclxIpbPU3F+YbwT4/1960V2j5/vudDDHA4HdJAedbgDxDeGckWrwY2U+tgRgNztkD9p/ZNG+Im7eqFNs8BHifob7tEKLyM6B2FZWRUbn0PqqfU657UiEbg4eB7yo3fMSI/l64Czxmphxsz4n/bkOIyIP01AKLZPYSJ3Gu8X8tmarNVy0O788/unfSwBXQyvJjiEcMM6lo6/vdXZCxu79UK3tRWGF/+WSDN9d7o7p8DQioodJ5EV387X11sZMTf1bzLxPfa2BmDZwfCg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1636803609; bh=JBIhaFzMp76eIITXMvFfmqZ0DMld1pzLEBCmQNi4kXf=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=NXb+5FDSIlFOaavaAZ3Tbifn1GVGyi390YiljeoVsOOZweKMotuKLBZq8Gd3/vpEGyWhEtAEXCusxoRJ0tadEx/oHmu3LaQT07LSz1CACMkcU9MODExq0YaHT5ijzioWqwm36k5dcbeQnszhMJllpSCEWUTRpHmmhQcJ1+gPEeQr2OjpKTXoPAvMB1gXV/0ILeCrRvMHwNQOPB9I+p1m3+CbF/3GIQ8XI1tphdNeZe/N7bQ3A34FyZPaG1LL42FAqmN+kh4XUgsBrHtqqeIXO3zxbwliAki0hpFTMZXoRQjXz/n/WnzFIxo7LGdkNKaoiSG9sCIPLh+7ee1q54tZwQ== X-YMail-OSG: L2s6dqsVM1kqGHxD4LskiVBbLtLJWbmAo1BiXYXIG3G09klk0jKflcfEV2mMkcQ Ud9_tMdimBqHasj0w3ztAnhnl6uN5TLEe3KMYq73vP00Hg3dvcCaeeSHxkPsE5A_XFyUlCpz.nNT YlrbWa4ZLDZDoBaA54jOtxfbwv.r0y49ACgeGGD.oRCzXuAYEs1EHk684vGo1_HlfyufcHngHKgS zcl44ysYdMj8xxaIdQ4V9iRnLTMvHbxAHd3_6rVG_DduOZVQVUIw_Uoi7TgfhbEf.BRneKDeeEOy iuyuQ9.Uxq.1U0HRaX98BtPmQpDBFL8.BL.3kbaV9rWfnh1DY92CQ2Vj1RFQ.JbIWDICPCDwHC74 4jXc6AIWYaXo99rVoXzSVmItCSvBN5qs0BkN1VRkzyY6aDldFWW4eWauSDzcyRsRAASTB4EeYSAt djxM.QcIJXOHUKswXheQNNo182cx2io2qFK38nISsVw.TYhmpY7Ujr8l76v3lQnD0VG6T5ltLGy9 GX8Phr5dJoa1su9tMSYYnLzyB4qPGQDQEfkUXFwxLPa0vN9g6RqUNR_f.p5DGP2mW1zozcJpDXMX R49MLmGBklP5zyB3lgvELVSrXcK.w1Qf8LE3NQm6yB1EaB45odlWnfreaq4oQ8vvjjiJXOIdvj_T 0WcNkH7MhUqZxxSr9A1BU2DFmwogl6K0NapxlElPxbsj3XN66MlzqTzf.lMoBGh1gbN6uXPJ6N_9 cE9iszeGPoUAWu2srcchV6wbShQQFuZdOkHB0KSDIz.P1i.N6GZR4pMlIx9jUFlgW5L9jwut.9RF Zo7j72S2Yjg6M4oVWIFekCj.vj8UrCBb2bqBNTBW9mjP.94WRtYIb6WqHAuXMY4HR_AUeiqg3Kyj ViIPLryh3w3yVLctsldEFp5XPSyK1bYxTdVguB_8MHLzQXiXq3Dq6jgd3uHGVR.3_XxKcjQs9RCt 59HHUhd4y0vO3fI_5CF.wD01SVXSugHlP.qrhZ_nAFbLDK9XSE1UI9nbcpCJMFXC4vDSgnkka2RU 4cny6fsgtZRQzvC6C9P6chcIrBpKkRyfwKWx6MKKAHQtQQr_VnW3IqV3iH5x6x9lfKzX_GL81Q14 l9hgHsEX7c6oiHbTt9Y6rqHcPtDNA1_tAPt.DUSy3vWO13JgwvHnLSWNd710UPN59LEaPpaf41RN fIWXPNEBditfFB_TxAqXsO9GSWLeocVCJTJh_5GMto7bSc6_Z5oSCiYj2pYxuFDEpBeRos4Fd3Pe 5wKAxAmVq4d_kedK5zBcVDsxD65jmyOMMtFn8_NtIOX2nGNUWGauJZ8VZmyKBViH2yIXxZeBGMZT PXZXiJD6d39ZDVBgE4L3UY6MaTvHJEmyZ_Sy.MXIl02wmCNBXWmANQuj64OhKz8j8pE3mQlxE09o 0P1WooqhPW4HyuOaaZNejYDvAEMdU987ael733ZI_f0de8RFdnNK8U3NegthNo8IHrqtBOIyIH2_ iZz8T9WvTY6Gq2UCkJcwhwCst5xPbfSjSGAUVeDw2wELXNBcjUOJ6dy2q0n9bDHkDOyV2biXnEuI SnhDXy8FuHm.S1KrXotwaFm4riOpu3dcln6gGiNTYPfXvW2eNLrd3rAF9oE1etRYQlrk2dLnqUrz .9Hzz2rQlS.I1slwSulrb0.TtBKuOdMf.Ug5raRQuY0RAc6jnhTSGOOI.eV9Ytmi1iyBH.U3cycp NPkIZgDszJFAyQAHNRqRL4kYeehZDELh_HgE8TqQPw6rGo96mNtGzb.S0U4aWZvwKLoJQyC3c7cB Hi2hZiHcJa2JePSDTAB8TiheQ3zXbGRAsyGYehapd72X4BoHfeR9.A3Uw4G993OcHQPB0WjJT_ag Fh2XGl0Lw_jJQRVsh0m8ansGTihEBzS4pifGnArJ9JG4RZ6P.WgH8BlCgTuwN6H0Hlv6l5nzCVss ryipWAp1bk4pPYRSbJuA5f6Ah3kDJDSoMBcNGBx_Bw3ApTUqph.uVylI8XwOxrFsWYNkKnjsTjKo Y1NXuHaPi9q1bkbdfDKEHRPJHF_KTiR51lu8L54jC3Nyeuj7b1E9vpuGISaZqRImdAbPcE3fTtLW 2kj2LNJSstz3O3n1nCEIHdW2rj2Y.Y3c5DBnknVqxNIG3e_CJhJjnnmvjAVhxUe_sv.L6fOG99fx PbKVVyLJ3OEM_seKwwPztRNxxAQACxQ.uBP3_dtdtyGekvMzqHk9qmck2jATRel20V_TsTPZZZUT 0Svmr60G7qxIZTkzqgI5Zinsm1IRGx4ME8HplJQ9cvU6.f.AKR8nCLZ9wvcpkySc_rcRSv7phekX iPFGYiq3PbnkaRPh579KlcwM- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sat, 13 Nov 2021 11:40:09 +0000 Received: by kubenode549.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID a3945325041a062581e571ec9e2baa43; Sat, 13 Nov 2021 11:40:07 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: FYI: aarch64 main [so: 14] system hung up with a large amount of memory in use (given the RAM+SWAP configuration) but lots of swap left Date: Sat, 13 Nov 2021 03:40:06 -0800 References: To: freebsd-current , "freebsd-arm@freebsd.org" In-Reply-To: Message-Id: <50D9DD1F-6949-412B-AE86-46E6F0129E8B@yahoo.com> X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Hrtmc2LpJz3m4W X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ZEvWZaO8; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.204:from]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.204:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-current X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Nov-13, at 03:20, Mark Millard wrote: > While attempting to see if I could repeat a bugzilla report in a > somewhat different context, I has the system hang up to the > point that ^C and ^Z did not work and ^T did not echo out what > would be expected for poudriere (or even the kernel backtrace). > I was able to escape to ddb. >=20 > The context was Cortex-A72 based aarch64 system using: >=20 > # poudriere jail -jmain-CA7 -i > Jail name: main-CA7 > Jail version: 14.0-CURRENT > Jail arch: arm.armv7 > Jail method: null > Jail mount: /usr/obj/DESTDIRs/main-CA7-poud > Jail fs: =20 > Jail updated: 2021-06-27 17:58:33 > Jail pkgbase: disabled >=20 > # uname -apKU > FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #18 = main-n250455-890cae197737-dirty: Thu Nov 4 13:43:17 PDT 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400040 1400040 >=20 > It is a non-debug build (but with symbols). >=20 > 16 cortex-A72 cores, 64 GiBytes RAM, root on ZFS, 251904Mi swap, > USE_TMPFS=3Dall in use. ALLOW_PARALLEL_JOBS=3D in use too. > (Mentioned only for context: I've no specific evidence if other > contexts would also have failed, say, USE+TMPFS=3D"data" or UFS.) >=20 > When I looked around at the db> prompts I noticed one > oddity (I'm no expert at such inspections): >=20 > db> show allchains > . . . > chain 92: > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > thread 100671 (pid 15928, make) is blocked on lockmgr 0%A%0EXCL > . . . (thousands of more instances of that line content, > I never found the last) . . . >=20 > My patched top (that reports some "maximum observed" (MaxObs???) > figures) was showing (having hung up with the system): >=20 > last pid: 18816; load averages: 10.11, 16.76, 18.73 MaxObs: 115.65, = 103.13, 96.36 = up 8+06:52:04 20:30:57 > 324 threads: 17 running, 305 sleeping, 2 waiting, 147 MaxObsRunning > CPU: 2.8% user, 0.0% nice, 97.1% system, 0.0% interrupt, 0.0% idle > Mem: 19044Ki Active, 331776B Inact, 73728B Laundry, 6950Mi Wired, = 69632B Buf, 558860Ki Free, 47709Mi MaxObsActive, 12556Mi MaxObsWired, = 59622Mi MaxObs(Act+Wir+Lndry) > ARC: 2005Mi Total, 623319Ki MFU, 654020Ki MRU, 2048Ki Anon, 27462Ki = Header, 745685Ki Other > 783741Ki Compressed, 3981Mi Uncompressed, 5.20:1 Ratio > Swap: 251904Mi Total, 101719Mi Used, 150185Mi Free, 40% Inuse, 3432Ki = In, 3064Ki Out, 101719Mi MaxObsUsed, 101737Mi = MaxObs(Act+Lndry+SwapUsed), 109816Mi MaxObs(Act+Wir+Lndry+SwapUsed) >=20 > (Based on the 20:30:57 time shown, it had been hung up for over > 2 hours when I got to it.) >=20 > There were no console messages. /var/log/messages had its > last message at 18:57:52. No out-of-swap or such > messages. >=20 >=20 > I did get a dump via the db> prompt. >=20 In retrying the poudriere-devel run expiriment I'm getting various builds that are generating multi-GiByte log files (and growing) that have lines like: thread 'rustc' panicked at 'capacity overflow', = library/alloc/src/raw_vec.rs:559:5 stack backtrace: note: Some details are omitted, run with `RUST_BACKTRACE=3Dfull` for a = verbose backtrace. error: internal compiler error: unexpected panic note: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report:=20 = https://github.com/rust-lang/rust/issues/new?labels=3DC-bug%2C+I-ICE%2C+T-= compiler&template=3Dice.md note: rustc 1.55.0 running on armv7-unknown-freebsd note: compiler flags: -C embed-bitcode=3Dno -C debuginfo=3D2 -C = linker=3Dcc --crate-type lib note: some of the compiler flags provided by cargo are hidden query stack during panic: #0 [trimmed_def_paths] calculating trimmed def paths #1 [lint_mod] linting module `transitions` #2 [analysis] running analysis passes on this crate end of query stack thread 'rustc' panicked at 'cannot panic during the backtrace function', = library/std/src/../../backtrace/src/lib.rs:147:13 stack backtrace: 0: 0x4710076c - ::fmt::h4428caffcb182c5b 1: 0x471c9d00 - core::fmt::write::h91f4a7678561fd61 2: 0x470e2180 - 3: 0x470ebd40 - 4: 0x470eb824 - 5: 0x41ed4848 - 6: 0x470ec690 - = std::panicking::rust_panic_with_hook::h6bc4b7e83060df25 7: 0x47100f0c - 8: 0x47100900 - 9: 0x470ec374 - . . . 65: 0x470ee71c - 66: 0x401361bc - 67: 0x40135cd8 - pthread_create 68: 0x40138b9c - pthread_peekjoin_np 69: 0x40138b9c - pthread_peekjoin_np 70: 0x40138b9c - pthread_peekjoin_np 71: 0x40138b9c - pthread_peekjoin_np 72: 0x40138b9c - pthread_peekjoin_np 73: 0x40138b9c - pthread_peekjoin_np . . . massive repitition of pthread_peekjoin_np lines . . . (I used USE_TMPFS=3D"data" to avoid tmpfs memory usage this time.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)