From nobody Wed Jul 24 13:07:39 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WTZ7N58vrz5RwcX; Wed, 24 Jul 2024 13:07:44 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020138.outbound.protection.outlook.com [52.101.61.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WTZ7N2Zqlz4mXR; Wed, 24 Jul 2024 13:07:44 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rJSPUKDXMRiVxiBDKLtK90OFLO78/bBUeTF3qIbbYIOt/L52w5okx4zvVG6f1hNYjwYsA9ltwm0f5VYGocjb3xVRztHyQ5DItlcWcBUzJMpRFftZ1RDG6qqhgFGQQC/kSoOD7YbTfdQ60yvuBhS0naLMBUrnkjn+1oWQ36Er9lZLHUsqvgrdz8BXwi7uhPg/nmkGyJq05/+WF1l+VJyH7eo373a5lsQOx6WN3lkVvnsPUcGCKF4Lf9OSZ7Qxu7NoTgdp9EYrYdywuFmsPE4PLq/wkp8+wkYr811aGK3KiGSogbia2bnr2qb/89mKFQRWIYD1EJMWdH062XnHq6eX9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s35BjKfbmEqCB0ldjE92Bb9PddiXCbtAiLJ2c9VSr9I=; b=JHy///AVpmjbnpnTGSj41tcCIEikw+8lETdyAfaIR3/gqlvf5Civ4l8DgkQ+8txv5e07NTXmgIBxVQB4Tymw8xNnolVoJwlF3SGzrm560/Lmfus/BmSQfu8AMbnvVXH3YE1/zPBKxeByNThc19Ddwia2MJmGYMCMs4i2FVEVy9o9jOZM7GaLBBNC9Ba+8N103pmlHHNFlLrskQyFFKjsOxhYv9FQYFH6poYs04YCUWLCKIO6tai5t0YJ5hl2v09Z7DDgGDHqxLao0tZPPToQRWoIr1n2JiTu7DmiutGVksK2p8MJZ41uC6/tW2lBp1HzDhrAeH+xzrnh32L3Xft78Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mit.edu; dmarc=pass action=none header.from=mit.edu; dkim=pass header.d=mit.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s35BjKfbmEqCB0ldjE92Bb9PddiXCbtAiLJ2c9VSr9I=; b=QXu3tQCvmYEdwv5XEa2z02SP7H6fQkNVARysvW155pJsornvT0/dWhJFxZLxQGOlM1zZWk/2+JuAiMMifpgxl9yzo8hfBXfILOPFgphmoDQwIpeM6aqo6aEcLTCxLed4lwIcfSUs9iu23gnMhG6nqvaq6w7GGPcrP9Q3Ajpvb6s= Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by SJ0PR01MB8059.prod.exchangelabs.com (2603:10b6:a03:4ea::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.20; Wed, 24 Jul 2024 13:07:40 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::bb39:d8c:f575:6b9d]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::bb39:d8c:f575:6b9d%6]) with mapi id 15.20.7784.017; Wed, 24 Jul 2024 13:07:39 +0000 From: John F Carr To: Konstantin Belousov CC: "mmel@freebsd.org" , Mark Millard , FreeBSD Current , "freebsd-arm@freebsd.org" Subject: Re: armv7-on-aarch64 stuck at urdlck Thread-Topic: armv7-on-aarch64 stuck at urdlck Thread-Index: AQHa2/xL/eBpVtdzuUaKV+38IPbVfLICobiAgAAf4ACAAC4tAIAABE8AgAAMwwCAACPzAIAAiccAgABEVICAABzIAIAAiPOAgAAoSACAAO5cgIAAAwWAgAAESQCAACZVgA== Date: Wed, 24 Jul 2024 13:07:39 +0000 Message-ID: References: <33251aa3-681f-4d17-afe9-953490afeaf0@gmail.com> <0DD19771-3AAB-469E-981B-1203F1C28233@yahoo.com> <6a969609-fa0e-419d-83d5-e4fcf0f6ec35@freebsd.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|SJ0PR01MB8059:EE_ x-ms-office365-filtering-correlation-id: fc4c91e8-51cb-4f5f-42bb-08dcabe198ce x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|366016|1800799024|376014|38070700018; x-microsoft-antispam-message-info: =?us-ascii?Q?p0SrxcG+91jZZXObxsNolStZ08rjzjGIoRqSvjrUpJRy5gYH7yjdiY+rVL65?= =?us-ascii?Q?MqpZZplJNdGaL5OE4f8rvYV52TkO/lg76EL4bo2HobcF3i6iL6EvvD8FBb+N?= =?us-ascii?Q?r3/lKSu0PxMSRfYREJdQvSJJ/aKj7lS5sSJqcE5B8pthDC2U4k7GQ66CpfFF?= =?us-ascii?Q?fInCFkf1EhngtRb/tRXM6zW38J/c3SvPqU4VL9K/Xu2wm6W0+AvRKuTdU2G0?= =?us-ascii?Q?YRYa0jG1jf4Bpj+mQOorOct1+91ip4m7QhrmwGA/r0TKE4f+2zSutKwP7Vyx?= =?us-ascii?Q?fkDMFmQnaBIoT+Zh8oYY7cdy2KJhKsya+kTdZHHcXw9FYnNSbMJDdFBRYwF/?= =?us-ascii?Q?mQtVlNy0KH0C7P3xd8Td2t2++UcCtGlFPdkF5+ZGbPorG156F5jiTqQuvtW4?= =?us-ascii?Q?JmTyM5T0xo36TEsVYcd2mX4FdM/7NQ5S6hA5sE04gP8bhJWRR0OKXt1cec1Q?= =?us-ascii?Q?7b18lSdvj+vH3c+F+LEy1NUcG4ylNQJdhHk75UMr20aMMAWTApotjLwpu1v0?= =?us-ascii?Q?SBoN08bCv8XQU/NrwGI/3g+aHd2KmvnijpUm+TFo8+1qbc+JRTzSalpSMYEr?= =?us-ascii?Q?Xf/TvnQEHWqNZ+4CnB1+479CcSwaviMTT0cF0cb7MgKnkHsgDDuSMOLFVO7X?= =?us-ascii?Q?Qu4DVL/Wm81P29gIiAsnOWP+T/+wKqb3+3fHOalrD0roK2aU8bZkZ59056EZ?= =?us-ascii?Q?F19U4Q4zZajOEVnMYIIEvpM2hsxRv9TNKRBbIrLwe14eoEb+tN7P57Vc21EY?= =?us-ascii?Q?j8mDVMRVOXy7hGJScgoiiHDjAiik0dWZ7iDvaqLRSuBEuPPLKLmKz0A5uQmh?= =?us-ascii?Q?OYAMVcOhWZjHPUJGdveeOLl+mYGy6gOau1UO7oSm0wPEJ3fb+ePT17Vo1cP/?= =?us-ascii?Q?LA3ibmsY9zrOVMhDd7bGCcablAn/104Okwc+orB/zGw9HZyhstlggP7UrMjo?= =?us-ascii?Q?Zk3LOOhB2zK7yN3FlewQGoSQdUuZ9sDQY6dRv+Kdyg1q8wAXde8gtHQ49buL?= =?us-ascii?Q?JuF5yCZQQtg3YCxUsch92whS+q1lyGO5KcMOoO8ZvL28iU7LFjLRe7bz4iK4?= =?us-ascii?Q?LevHeSXvhCti5gV2okJxdnLW7Ne8wdVUVgGn5fAEJVJy+gwtdvl7Ams1pakx?= =?us-ascii?Q?3vDb5+dU7OGrbEsAJH/U7Gh5i0PBACXZkJLY4e2F5832cav/Wb5XoHR5oB1V?= =?us-ascii?Q?N9YNkUONz1K1Wj0VVxsunvOv7Fat8qDVV8NKLmc8kqT/jLYTJvPhr4bKSN2g?= =?us-ascii?Q?N5Ymjc1XUu1BKwo93JclW17wQzPV4d5pMOatIpURx03WdnR1Wkkw6M0oY5ow?= =?us-ascii?Q?j4OmSlTUeCB08za2omlrtVe8wYpnl+bA/FWnLKu3+roKAGEkqIDOvlTlNO1j?= =?us-ascii?Q?1ZEQxHI=3D?= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(38070700018);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rnWxWt4TkFJ5I2dRYhcgkujMVMcAaeHiEce2jLTIiX/XBDWbj2jcbdiNZngJ?= =?us-ascii?Q?fsKYDMxr6iJtZTKu+tJ8MeAwNC6cc/oB0Reed46h4tOKNvLvWMbPhE+Ms78m?= =?us-ascii?Q?OcKnhY7jmzo0YVSFiHfmZLrWEtLr3UwZc07sXHi49mX2odpnu724Et7FnN+T?= =?us-ascii?Q?hcZPS+sbSN3KND9QRMiaqmet7aezd9wcKg16Km+3Fp8Y6S9q9Ki9WCipdmvl?= =?us-ascii?Q?v3QHaZ5GAQOFLd9JqhghZJj6wgaDyPL7kVndsdSdDChzhfQ/U661gCKmbFuc?= =?us-ascii?Q?A76tKpHoA/Vnqo14QgJJf5bFiuoShNGNPEwaEbaRR1wMUj9icqAL2U3Cbybu?= =?us-ascii?Q?DhYt9mXQAp4VHB0zgvz4Ruqa2wVnTyUImRiSNea9FS3vmzAluroh3QMze4k8?= =?us-ascii?Q?UvpQC73iBSi21CPro/Jb7bqPtzrUvf0SbSjiNDS3vE41oZPLXDqLkPuU7WUb?= =?us-ascii?Q?ArQ2OWKWNN70ARpyNHDbCddDeIUttyxs/7iFcp3RlNLZDUAkkD0b4zikS0im?= =?us-ascii?Q?fUWV98v3QfLixp0UrDld2yzjPN687N9pHEpUIbUjGSa1CWHbTc6fbs9VWL3V?= =?us-ascii?Q?tVD78go78krAVZ4pVdY8p9wjwQozoBiYwe6w0q7Al1XiID5Hy30r3kSkRJ8U?= =?us-ascii?Q?QLUP2qvUrR7RExr8FfZii5g0cOcy6UKppGo6fRJ4rsAi/NljWmBtlUjUMKC7?= =?us-ascii?Q?zzo068ra9XD14oOTMz8UWpNPqmskFAS7NUWxjDa/iGn4AfzhuG7QiPx+PZA2?= =?us-ascii?Q?vjReVOKA1kSEzak9qZ506zsr9zpWs46qvi6wFpMi97FYYmbxuoK5brrOwN0o?= =?us-ascii?Q?Rv/fzFCXnikQ+OwO0YjSUz023TEJUZd0nhs0Thn2/R0S8UQYmWBJ4p+21ijz?= =?us-ascii?Q?9ND0gf647Ulxl701fVcYezCLLoSIiIoewLy2mp6HVnZ5j0am6Tlph7pblpqs?= =?us-ascii?Q?BvT2QUxH2g+N1d/w7k63zr709JpEeSfWKUsizVZ/r9hbpweJry0ux2vI0uIC?= =?us-ascii?Q?IyxHuG8E8cOL4WuzWO4bW/aGB4zFV38fRSFFiUnD2DPhlyvuQOfXmbsIQ0uc?= =?us-ascii?Q?OyxWLFksBaThof17zUJt+m/4BCv3zC4w01pzW0mnQrhSB9D8Yby/KuPJ4/m9?= =?us-ascii?Q?tp/yQHiizTMPmYN1NkPoZArIndii8LgaABrWfKUq2ZPiPj+J+zYGsTzncjxz?= =?us-ascii?Q?V3rZmy96FRmU8uujCv/5Mvx+PlW8JdmivoKhWw/XlM5FdmELeO+tU0GI0v08?= =?us-ascii?Q?1EqTAAB5rhyRttWB8r3ayUOzcATpbvlZ6+92KRGhY7ihQf2QoW/kLbcLaaJo?= =?us-ascii?Q?qMUhsPAPv1cP42Gg8lpGvcJT9ociAqTFm9+nSQ5/UEZGedF/rVLpcvalkZAv?= =?us-ascii?Q?q8IbTwYBonSXyJmlydJ8Ya49S4yNketq1E7/FaecebvLPe0W33EkMXHBzKss?= =?us-ascii?Q?t3B8KTzD/uJKy3GVkHM8BoxsQKiTAYTGwywGANJ7c5XKtljXvGiYKsBEjhl+?= =?us-ascii?Q?X8lvrVBIxXkFhREJqP+NXlsUR0LLDvAWJ7ksxb0dnx78vAT8xmauRB6LFSyh?= =?us-ascii?Q?FTk6oqcKW9g0v9NwSrRBU+teiOmsyorgd33V5ChAT3XsAfwpbhVH3PMJuFXD?= =?us-ascii?Q?VB2FT0Yhhsm9GYQaFPWSqx7Aii/UrWcuMt/XsnoDABoN?= Content-Type: text/plain; charset="us-ascii" Content-ID: <8279D1131CCE394C9C9113200642E6B5@prod.exchangelabs.com> Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org MIME-Version: 1.0 X-OriginatorOrg: mit.edu X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: fc4c91e8-51cb-4f5f-42bb-08dcabe198ce X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jul 2024 13:07:39.8919 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rP9/+Plrxcs84oX5ZgADBLRZXRJE5yuC2CsqPefTwCADjVHo8rumMW3GOMWlp9d2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB8059 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US] X-Rspamd-Queue-Id: 4WTZ7N2Zqlz4mXR > On Jul 24, 2024, at 06:50, Konstantin Belousov wrote: >=20 > On Wed, Jul 24, 2024 at 12:34:57PM +0200, mmel@freebsd.org wrote: >>=20 >>=20 >> On 24.07.2024 12:24, Konstantin Belousov wrote: >>> On Tue, Jul 23, 2024 at 08:11:13PM +0000, John F Carr wrote: >>>> On Jul 23, 2024, at 13:46, Michal Meloun wro= te: >>>>>=20 >>>>> On 23.07.2024 11:36, Konstantin Belousov wrote: >>>>>> On Tue, Jul 23, 2024 at 09:53:41AM +0200, Michal Meloun wrote: >>>>>>> The good news is that I'm finally able to generate a working/lockin= g >>>>>>> test case. The culprit (at least for me) is if "-mcpu" is used whe= n >>>>>>> compiling libthr (e.g. indirectly injected via CPUTYPE in /etc/make= .conf). >>>>>>> If it is not used, libthr is broken (regardless of -O level or debu= g/normal >>>>>>> build), but -mcpu=3Dcortex-a15 will always produce a working libthr= . >>>>>> I think this is very significant progress. >>>>>> Do you plan to drill down more to see what is going on? >>>>>=20 >>>>> So the problem is now clear, and I fear it may apply to other archite= ctures as well. >>>>> dlopen_object() (from rtld_elf), >>>>> https://cgit.freebsd.org/src/tree/libexec/rtld-elf/rtld.c#n3766, >>>>> holds the rtld_bind_lock write lock for almost the entire time a new = library is loaded. >>>>> If the code uses a yet unresolved symbol to load the library, the rtl= _bind() function attempts to get read lock of rtld_bind_lock and a deadloc= k occurs. >>>>>=20 >>>>> In this case, it round_up() in _thr_stack_fix_protection, >>>>> https://cgit.freebsd.org/src/tree/lib/libthr/thread/thr_stack.c#n136. >>>>> Issued by __aeabi_uidiv (since not all armv7 processors support HW di= vide). >>>>>=20 >>>>> Unfortunately, I'm not sure how to fix it. The compiler can emit __a= eabi_<> in any place, and I'm not sure if it can resolve all the symbols us= ed by rtld_eld and libthr beforehand. >>>>>=20 >>>>>=20 >>>>> Michal >>>>>=20 >>>>=20 >>>> In this case (but not for all _aeabi_ functions) we can avoid division >>>> as long as page size is a power of 2. >>>>=20 >>>> The function is >>>>=20 >>>> static inline size_t >>>> round_up(size_t size) >>>> { >>>> if (size % _thr_page_size !=3D 0) >>>> size =3D ((size / _thr_page_size) + 1) * >>>> _thr_page_size; >>>> return size; >>>> } >>>>=20 >>>> The body can be condensed to >>>>=20 >>>> return (size + _thr_page_size - 1) & ~(_thr_page_size - 1); >>>>=20 >>>> This is shorter in both lines of code and instruction bytes. >>>=20 >>> Lets not allow this to be lost. Could anybody confirm that the patch >>> below fixes the issue? >>>=20 >>> commit d560f4f6690a48476565278fd07ca131bf4eeb3c >>> Author: Konstantin Belousov >>> Date: Wed Jul 24 13:17:55 2024 +0300 >>>=20 >>> rtld: avoid division in __thr_map_stacks_exec() >>> The function is called by rtld with the rtld bind lock write-locked= , >>> when fixing the stack permission during dso load. Not every ARMv7 = CPU >>> supports the div, which causes the recursive entry into rtld to res= olve >>> the __aeabi_uidiv symbol, causing self-lock. >>> Workaround the problem by using roundup2() instead of open-coding l= ess >>> efficient formula. >>> Diagnosed by: mmel >>> Based on submission by: John F Carr >>> Sponsored by: The FreeBSD Foundation >>> MFC after: 1 week >>>=20 > Just realized that it is wrong. Stack size is user-controlled and it doe= s > not need to be power of two. Your change is correct. _thr_page_size is set to getpagesize(), which is a power of 2. The call to roundup2 takes a user-provided size and rounds it up to a multiple of the system page size. I tested the change and it works. My change also works and should compile to identical code. I forgot there was a standard function to do the rounding. > For final resolving of deadlocks, after a full day of digging, I'm very m= uch >> incline of adding -znow to the linker flags for libthr.so (and maybe al= so >> for ld-elf.so). The runtime cost of resolving all symbols at startup is = very >> low. Direct pre-solving in _thr_rtld_init() is problematic for the _aeab= i_* >> symbols, since they don't have an official C prototypes, and some are no= t >> compatible with C calling conventions. > I do not like it. `-z now' changes (breaks) the ABI and makes some symbol= s > not preemtible. >=20 > In the worst case, we would need a call to the asm routine which causes t= he > resolution of the _eabi_* symbols on arm. >=20 It would also be possible to link libthr with libgcc.a and use a linker map to hide the _eabi_ symbols.