From nobody Mon Sep 12 22:08:03 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MRLMJ3hy4z4cSKp for ; Mon, 12 Sep 2022 22:08:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-55.consmr.mail.gq1.yahoo.com (sonic316-55.consmr.mail.gq1.yahoo.com [98.137.69.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4MRLMH3K9Zz3nPY for ; Mon, 12 Sep 2022 22:08:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1663020489; bh=Juet28JHQoTk7pUt6F14Xiwobbzqd1dn7xB7vgJkxsc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=KH5KpTB/zF2zuCbwJtc7XGwxbLlrqpFV/ePDyMcxuYgG8+vCZirb1Fa76j5nkpefEP4yzCRhRZrwzNHX3wsUDs/EKYx8DOK8vk+0y3a2BSxpO0o+0O1B4CXlFRNwp/ygZrt6QxZyyUH4zN2GB+T3ZR5JE6EHdr21b7vE592wrCqrOFi0WWlGe4YlELGWzSrn6/0l0KYSoLngVCDmJpbOCO+V+Hd+IqsksuXzJDE7QEaJ4SYE8PUjvy+ii7Lg8iQOAOzpOJHEN/lH+UiCeC+KDbqwunBwXam74uM6ERH7VcOQ4REHGGdKqmEphTyTv9TiG/Dsa9onFm2/w+DtwYBQbw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1663020489; bh=9MF9J5Fwq2IggiN/dpBKn/At/qon/3irl8EmVs5ZwLC=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=SBee2E/msP3WKuvcLVXhtocCvkzyN4DpymGRUymSizSrhrr7k6MpxQNcLTQzfdmpV/6Am35+NnKQAaE5u4ifrBjL/EtotEqwky+xvRiVArldcJJV/zR/cK2G86Q3IXWjQyOvOHaNr2MZ45tRM+pXIlBPZx2Mo312N3XXZz0FTshrxMiI5HlTkayoSFczyWGPUqNiV/wpMnT/8tBVgXV9jXrvixZHCm0LHpPYQmgzIqPDGwfZjDUALryDrxL+zkNAHbhv+din7ybheUCT1Pof0ekTAJHlvPTWPha8A5n99K0dSfTmZzdOwmRCYaCn4S6Ko6gX2caNdFBotDLQo6+rJw== X-YMail-OSG: k1FghOsVM1k_pE9Cr1T3r55xkXQTlHmc2MQ7IQ2ci41yJB0cnKuuJiOlwEgjgtP ajLGYM.TGyQkwBoMk178Wf7X02x.IpCzPNXi.nTayVtVPSamzILlDA0K934vFTrm4T7yqqV.GUIN rN2TZaMSGpr1fCnuT8GQs6jqc0VJtbpd8lHOa5JeDcHjwggMc5J655PkLkfh889Gx9UX4qmVQVJ4 54lFdjLrdFxckHDOfE.kS0BQvdtHphJu.UKQth1yJTb8xBK_Z5wYPOMmp2173nJ9aaTRmLZ0CyLl rkdps0sLvoI2tRyjF.3WHOZSqz4Z64iPgGwrR0DPDh3WWoz.JiLwjvoYlAfqHC.96_eQryM8yLng LEuXxnHrdbJP5_CSCAgtTSg8eGUiF9Ry8aDBz7.b3Q4KOsS4cWxOn6VhbxGsTKe9zli54JZSMGMm El4QWu0NsFAhAuCs8sTSt9I7Z9y0LmgABzDVZN8hG4aPqmrm8yqzxyvxMnNccOslJoE6z0EjUhG7 cQUkC0SM7n.55Bm_5w4czLI80gQmhImw2g.kT7uO1hJ6L0jvEa4f64kqdbqLBybL7VWFjlhEvnOm 0VCBjXuEw5HyAkcxBHH5wtYwNGGce4XOIMMCEFHCraEcDzCs0A3o8fDjYuw8uNnZsxwhDvLAgmN4 5ALvimLknO.j9fZaR8Mke2zWw_mGuuec7iq1eWGYCpY5HgqrzpCXYn9hn_aQQGCyAe0guvLMaE25 vd5jVgmvoBuz8X5NnxHIFYCmfWkXbRLXDBzrXJMe0hLghgQVj7Qm9YfTHUSRKDDWU8.tpeN1.P3j 55x5ofaNzauPwFglRcBDWg2UCpPkabSRUKPytoVPk2okj2dZZcKyHvQBerBZAVhN78LIVfi8sbWQ EvO0fDMO72bem0B.Q3lKpijqHgDXSNebriiu5YaVO2H494AEePBaWXo.Pi7JCSIlkjPPPaL7faSm C3_N2Cnc2YMW6D7CMgW8ApmTc41jpAIgtlmsZhivYKTII3oFxYDpKBehHsoJfxqPOmcK8JKMGZ12 VDg10Z_UIYi.B63ATQ_rvxCatT301.cZ8.EUzC2TNlwCqvpG8K3RTbDCIeRZtUJktbDRW76eCGKo euwadv7vjiHooqoMAOYrMyIIQW298I4AWKjUjb_8khYV4jctY5Y99bl8GXM4XO.x2xhxj.fsCUxk M3ACOR90vond1taVmN89l6cXoeqwwQ6KoSY5RYo8Jo3rakHNUjEXCFN5e8PXfpwQYA8GDH99dGEl dUbr3beHGSAML9v8gerItLxI7gockVr4BbJ.pgwbSajs3dMo5_KIrUdPBxcDy_8ZYvhLU7P2Mm3_ 95gzfNNYRQ8.uezNRlv7VLQgpiaPjh9OkxVz1QuUTTM3EnDcgqKs3uLcEMY2a5IlsO52doVUK1Ht U4yprxLiKtXgWrHdFXO9fSx5HN8G2PuvMNIfr0ociDYyeRjpeFu7PI_QRrPEU8VGAXjrjj3jkpEY Go0HfOS6d_ZTLKsIPOjo8EyEi0vVrMt9ZQYYEE6YVzZj6d4nTt6E.YBNgq0TEPfziiGbU7_a4s3U 4TuATJGbnWkH57n0HBOQfAp2SHsWskD_9gScfA34D2iXnswWaptRf6tGbrz49KDFy71zMRraZn12 dWeG2ddU9pXKx6lJzQ9b8_00uup1R36rx1C.VqYaJu2xwTBxSWfV4a6v9OtsrNB4XPmoXFK3sJ2r jHYZvZrow4lTen7oEOzcnrleUEh.qqfNevYV4iBS74A2aB2xNwXAXEd5zhIvXZgUnAVtqnl9skqP hH1m.veGPWoJJg84hdSGqFYjBhbZRE7_.t43pNPRYcwlwbxBVtCFDwZiOXJodDa3dpprcKTnsK99 1AqWoeg4EC6gks6n6MklY7HkXuMkb8GSAy4DEBf0kKozNmSAjJoLNDTaIYlt5zDUitzwXu2ik8hX 95bRc5g3QZsn5n36KfMw_qmvlKgXkWmZkcuBDM5RJQcShQYXB8G1H.29xAPq9I2sZatHXquO94BQ Noyflevo8zejEzIC.d5Mu4RpKkkMYx.cZvoQbO_9uPALz9Zn9J3lwB0eA5uFq7I0.wX.MCYluYtn 9wgS6YLOfChHoCYm8GlKFASYapXU50iyV5GqzpJ1X.ZrSpKpG8Gf.6joF_a7L9rDGVplr7fVQ9US _mEZNSXoJPYtfBZscxx0K9erP7eOp0Tfcm.4toJiyx7rKrQwY7ad4RBYXddhy6avKvlHcst1EPfi 0856v4a9066Nn9fk.MbbSam0Th6xC2lWd0PeLYFQyPgETgqqzrJTY54xBvTZ3NAMYSr3.iCGVg66 NM1XwY9M- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Mon, 12 Sep 2022 22:08:09 +0000 Received: by hermes--production-gq1-5499fdd576-49dgt (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 84fd9d53cc1b48dc6ce704638f11ed39; Mon, 12 Sep 2022 22:08:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) From: Mark Millard In-Reply-To: <86czc0eotc.fsf@peasant.tower.home> Date: Mon, 12 Sep 2022 15:08:03 -0700 Cc: bob prohaska , Mark Johnston , Andrew Turner , Ronald Klop , freebsd-arm , freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <9E1552DB-4A65-4DFF-BC79-CFE045ECF972@yahoo.com> References: <1800459695.1.1646649539521@mailrelay> <132978150.92.1646660769467@mailrelay> <3374E0F8-D712-4ED0-A62B-B6924FC8A5E2@fubar.geek.nz> <20220308154204.GA37265@www.zefox.net> <86czc0eotc.fsf@peasant.tower.home> To: Dmitry Salychev X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4MRLMH3K9Zz3nPY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="KH5KpTB/"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.99)[-0.993]; NEURAL_HAM_MEDIUM(-0.99)[-0.993]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.31:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.31:from] X-ThisMailContainsUnwantedMimeParts: N On 2022-Sep-12, at 05:10, Dmitry Salychev wrote: > > Hi, >=20 > It seems that the recent 14-CURRENT/aarch64 (866e021) with DPAA2 = drivers > panics under network throughtput stress test in random places 3 of your examples get a signal handler called at the exact same instruction: #6 0xffff0000004ced5c in witness_lock The parameters vary, as do the callers: #7 0xffff00000043a3a8 in __mtx_lock_flags (twice) vs. #7 0xffff00000047d4ec in callout_lock (once) Showing one more level, where all are distinct: #8 0xffff0000007d60a8 in dpaa2_swp_enq_mult = (swp=3Dswp@entry=3D0xffffa0000056ca00, ed=3Ded@entry=3D0xffff0000bcda2c70,= fd=3Dfd@entry=3D0xffff0000bcda2df8, = flags=3Dflags@entry=3D0xffff0000bcda2c6c, frames_n=3Dframes_n@entry=3D1) = at /usr/src/sys/dev/dpaa2/dpaa2_swp.c:795 vs. #8 0xffff000000508f54 in soreceive_generic (so=3D0xffff00011d2c2200, = psa=3D0x0, uio=3D, mp0=3D, controlp=3D0x0, = flagsp=3D) at /usr/src/sys/kern/uipc_socket.c:2240 vs. #8 callout_reset_sbt_on (c=3D0xffff0001121792c0, sbt=3D, = prec=3D, ftn=3D0xffff00000047d4ec = , arg=3D0xffff000112179000, cpu=3D0, = flags=3D256) at /usr/src/sys/kern/kern_timeout.c:962 (no address shown) Perhaps looking at what the code at 0xffff0000004ced5c (and before) is doing with what kinds of data would be useful compared to the less frequent example signal handler invocations. It is common to all 3 call-chains above. If dumps for them are around, more than the code might be able to be looked into. > with > unknown kernel exception 0 esr_el1 2000000 on Ten64 board (based on > NXP's LS1088A, Cortex-A53), but the same code doesn't panic on = HoneyComb > (NXP LX2160A, Cortex-A72) even after ~10h long tests. >=20 > I've gathered some stack backtraces from ddb and kgdb (attached). > Panic itself can easily be reproduced after several minutes from the > start of the test. I've tried to change PCPU_PTR macro to use get_pcpu > again (as discussed in the thread earlier), but it didn't help. >=20 > If you want to get your hands dirty, DPAA2 stuff I'm using is at > https://github.com/mcusim/freebsd-src/tree/lx2160acex7-exp (branch is > lx2160acex7-exp!) >=20 > Any ideas or places to check would be really helpful. =3D=3D=3D Mark Millard marklmi at yahoo.com