From nobody Mon Mar 07 18:03:51 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3A98F19F54CA for ; Mon, 7 Mar 2022 18:04:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KC5ts0mDZz3D71 for ; Mon, 7 Mar 2022 18:04:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1646676237; bh=QNJmAQMRxgmElsmewRjV6HwzLY6BrLjlqtFuBV2iLXY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=G18lIaRC0WkvovsONRYxNO6qaxtOIkNxt6nVsb3zuk29jBwj44ukb8P8DTh3q+Cyj2I2czWjkTRhHi3NUOoQv6ihtfBcNa6GSfewkdDZpw8lwnxcUnHTczfQxsjrpqv4S+Uy9eoXCchFDSODkzyBRjhBCY4wXQjQvc6p8sLpTqmp4KqkM007Hgxn8JUhPiQqyfnrflqjSLgrN2jOBKwDrPmGpmS8117riUicAv7n7E8cL60Ujqz3wlz5R2wWcHlm3PTCqi/abXGrGDxnTO/CrzYqlOcC32SRZPvMNWNv7khc9LQqMYwqDCxXllIBjaM96dVBNvW2HsSIztxLycjyww== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1646676237; bh=D3B0Zj4SUlq3Qcq4JJd0joW0jjk5s2cohbbX9zsEFNu=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=FxoXkzSrJIgquaSw/URh2GneEDRbsyHYd4+6hQ4lQoqeX47OVk6WzRJUk/RciQd98bm+s3D+yj0wV68Gp5ccEFWRPUF07hfHyN+3UyT1xsqvzK+x4kkKNoqTFvhPArsBHIGE3gtLY3U9TLZAoFahWWHoxS0b8Icgl2GdRz6Wn5AQpiixDKPt1ciu3UjS+joLkFh58mbfqVK3kGU2yPjVvqD57y/2JXFLLcaDKHalTeAknwUZ06eut2bEztaiVPNZUw5j5MUKOuKCBbPVoxlhHTYw1bBZD0rOn8PsiaDPSxrbRQPBKFsgYoth2RyoPYcFGgqMhjhQl5Uf5UTQOqHrTw== X-YMail-OSG: f0XSjhAVM1ntBc.DAjrpYvHF8g7bQAQyqUXDaPNvHjX0cpGrck4Ss1wEdNEWdWg E76LC0CMwrzz7xQ8zgGMm6ozXOxHvaSq25HI4arbyvFJr6qeG.UMnQznP1iqu2q_W6XWyYlET7Tx cXRPF_1sZXr1GvI9xS6QMngRuaYuSemRHNd_BSSmy98sRvYskmX08cPq6tgz.WsIskxUT7LJa_XE jyfn0MjIc4sLi7S..DdbUUFrFQGE48pLEuRCnKQcoPNnP2SmPru.L14W1dG1ENq.o1FDUbe3gKro Bh4gAxVd23EqncWBulqaPa6A6GVia_4__LrHH1WMzo0vNcA1uZlH7gB59al73nng3kQgxF2jRgMl UT6zblYD3NcYp6xh.zw3b.hBM3h0CCeufoBI1X83IBCZy1SUMrtUIE7pj5VLvtfyEsBtP18X0_sZ 1LNhNFaA8CJIidEleS0a98xEqZvNwiw_KJAe_vV2dRuB4JrzHyOUMQUN0FICV97hRKEozaX6Ea7W h79A5E0xBUVEaZlIkcAdjyUsyBQDmjg.PoQKm3sN78lhP25C3lgiv.wBxY.YzWdSM7DGS1Xc2wQD Qew8vVGgzTHVz8JgdNidrwiQEbo6IiJI1P85bO0NTA2Fo01y0dS7G_3AEN6iFQPFAwsKCWykIpIy CfFUOEWpdMqch_ARfcW5X_ffTL2g4iD_ZCbIjyDS2j7iFRfPx_2QC_7Cabr6btd_UkVVkX3f5Cpl 5XiViVBlrkbPZ4_.UoPL72XbZjJqyVUub3jeVLUsF5X53._jVIbKjE1G5bbqRKK1KQJIvoYZ8U2a kFZe6tyPgSWmhcVBs6ru4CphT1tjmiRgZYNfyPZYUBXwk7F0AOKfxXmL4KZX.PfxOoUqzCZW4uIw IM41JrnIB1PTiWVT6WS1.0php59gJlHGz.z0KN2qfgDA8iIOxfBuo.i_NH6nikr4SLMdbvCtv7KJ 1VGWiXBhsUTzJeSNQL1rCcI9h2qleVaibpp4Pc9HnyS6y_dNLbNpNv_Gsv8vYkL4g49bhxHj2X5b EcNiN6rCZHdGQAvLqf4A6SSMiFlnSO.dLFT8QvQ5pOY10NsKSAZFXUnrAACaRDrZQ8PzYNPXdlBq nzWErbuXt7n9xdmy6FPVEOr1Dnhl8Cdl7vbpjbR.bZn5W_cUcmU9z4UeGPUoJRRHBTxBcti8wUVp 3rj_dVQCURlLkRNv57IgfmA4ZeHEl3BbCYnyNFrXbsFHnoYY0BgHINjV_zWNNROObQyKEryfcHdk MS2Uy.jnpvnNK2mbwtNTBflsUM0p9BzI.VBYcTih9kxJSuOa.Exj08MIxObJsushj5.uy0MGPpBv cSGpYcd4D9jiumShDWCUMDGg5hSG.ioCZU7h3rc.OqClOo6r2xM5NQx7QI9_hg13yrtAKsv0Kl7j GPtxgeNrIJ8kskLEpkq8NH_907QK_Bc9YRSPNwJ5jSV2XCLYtvqoQ_WawILeCvVMyW41eEWU90YI qSJptWb5JwdnD6CeYl5SX0n10D6Wx0xo12zcelN96ZFD.a3M7ql_QlvCkhI2purGUBFnAZZvc1BO hZUI3y0KSTmLGYPqoseQRSqOHW1KHs4TaXQgyThdk8dweQSDVtWttm9S8KMqDTmfyJN15rhQRvtl sNhWNZZuD13KF0NyQrVRE..n8J_36OiF2twwpaW0NjaD2VDDYzx4fqXrTimXTC9A7LayW0VYLo8m NvvBmNvZomV0RuYhTNmUv6ngbhPxUp1F5LejWh2DDsJ0i6g7DJe_IimoDsL2RUBS_vRV5bOgNAGK .H4dLb3n9PCeb1ml7ZjaKWFLCkBd0C3ysbsug5HiRk9KQXJJKA5deg4qdF8.J7NyM9ReEzJrq6UU iWsZGniT7mjgreGLmM6mi.lbnvKeJrgHzIVVCC7z_dMLiAUxfkRNTWh6uo1GFmPH8Q0okKnjYPeQ OsDImabpVwq790f23IMRreoZ07.QuyTVDs7E5vPjCoFoa3R9xmAzfTB5CY1HTQkEStvj3n8rpBAX GAQhrkhlkeRFevnJB0_wH5Ut50rHhGQjTPhBgmcpRnm6d8bN_rI.PdNXnpul8Ekf7jGgHanvtKW4 mP5SC_d1jwQNGTy4goiMri4yYuTn.tWPVNcMeUq5sMjPc2y4O3qFL0k4zZMF4Ciaa084PK_rXJNe Pvq0TZJF80EkL3W7_5LDtC1F54cnz9.Lf71nBo1Cpu.Fd9MkgNVwzGFmGkzP8Xd2er.14dpePSRD oBbpplsw2o1_dwhAdgg7yOL3z29Y5NJmctztjJfYZv_t2jc_pjKF7LRnNMQpvribcQRd5f9bjnLm Vc1gLEf8cIIaeEhHfgSPN0huC7JX6iDxGViwHQAAUVUDtg6agrZAmPf4gSn910Jz964vUE9NOAuS jwgAOSQ_EMRGaEBg- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 7 Mar 2022 18:03:57 +0000 Received: by kubenode518.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID beef9abb4f3e81af87e938ac5564e7ad; Mon, 07 Mar 2022 18:03:53 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) From: Mark Millard In-Reply-To: Date: Mon, 7 Mar 2022 10:03:51 -0800 Cc: Andrew Turner , Ronald Klop , bob prohaska , Free BSD , freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <1800459695.1.1646649539521@mailrelay> <132978150.92.1646660769467@mailrelay> <3374E0F8-D712-4ED0-A62B-B6924FC8A5E2@fubar.geek.nz> To: Mark Johnston , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4KC5ts0mDZz3D71 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=G18lIaRC; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Mar-7, at 08:45, Mark Johnston wrote: > On Mon, Mar 07, 2022 at 04:25:22PM +0000, Andrew Turner wrote: >>=20 >>> On 7 Mar 2022, at 15:13, Mark Johnston wrote: >>> ... >>> A (the?) problem is that the compiler is treating "pc" as an alias >>> for x18, but the rmlock code assumes that the pcpu pointer is loaded >>> once, as it dereferences "pc" outside of the critical section. On >>> arm64, if a context switch occurs between the store at _rm_rlock+144 = and >>> the load at +152, and the thread is migrated to another CPU, then = we'll >>> end up using the wrong CPU ID in the rm->rm_writecpus test. >>>=20 >>> I suspect the problem is unique to arm64 as its get_pcpu() >>> implementation is different from the others in that it doesn't use >>> volatile-qualified inline assembly. This has been the case since >>> = https://cgit.freebsd.org/src/commit/?id=3D63c858a04d56529eddbddf85ad04fc8e= 99e73762 = >>> . >>>=20 >>> I haven't been able to reproduce any crashes running poudriere in an >>> arm64 AWS instance, though. Could you please try the patch below = and >>> confirm whether it fixes your panics? I verified that the apparent >>> problem described above is gone with the patch. >>=20 >> Alternatively (or additionally) we could do something like the = following. There are only a few MI users of get_pcpu with the main place = being in rm locks. >>=20 >> diff --git a/sys/arm64/include/pcpu.h b/sys/arm64/include/pcpu.h >> index 09f6361c651c..59b890e5c2ea 100644 >> --- a/sys/arm64/include/pcpu.h >> +++ b/sys/arm64/include/pcpu.h >> @@ -58,7 +58,14 @@ struct pcpu; >>=20 >> register struct pcpu *pcpup __asm ("x18"); >>=20 >> -#define get_pcpu() pcpup >> +static inline struct pcpu * >> +get_pcpu(void) >> +{ >> + struct pcpu *pcpu; >> + >> + __asm __volatile("mov %0, x18" : "=3D&r"(pcpu)); >> + return (pcpu); >> +} >>=20 >> static inline struct thread * >> get_curthread(void) >=20 > Indeed, I think this is probably the best solution. Is this just partially reverting: https://cgit.freebsd.org/src/commit/?id=3D63c858a04d56 If so, there might need to be comments about why the updated code is as it will be. Looks like stable/13 picked up sensitivity to the get_pcpu details in rmlock in: https://cgit.freebsd.org/src/commit/?h=3Dstable/13&id=3D543157870da5 (a 2022-03-04 commit) and stable/13 also has the get_pcpu misdefinition in: = https://cgit.freebsd.org/src/commit/sys/arm64/include/pcpu.h?h=3Dstable/13= &id=3D63c858a04d56 . So an MFC would be appropriate in order for aarch64 to be reliable for any variations in get_pcpu in stable/13 (and for 13.1 to be so as well). =3D=3D=3D Mark Millard marklmi at yahoo.com