From nobody Fri Jun 28 06:26:34 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W9QSp0CXwz5PPNq for ; Fri, 28 Jun 2024 06:26:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4W9QSn0JlDz4T2W for ; Fri, 28 Jun 2024 06:26:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=HmvQtNse; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1719556006; bh=0eWF/eRhn84OnNCM1vcQVTAB8noaE0cYCvPB6In4agU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=HmvQtNseLkMdWmRWEd/fo/MfLbLjY2iXj8VypdVXX9sQthyYkCbbXUL/aRkqDkyq5BI8IWC3vuiKfUcCnVyG4KAo7wIXrWRoLOXgfnQ5cMyNjrJozVw+ht14lqKMiXBN/7FfdSS7fyA6hOM1D6hOCtKJrc+LUAR7mcozttCxvt7FWotWsaO6GTLRiVXkO15ymq5GwWe+TRsza+HuaeTCQ5HXltoSKuo5ECnqg5xjqxphUl/2JmKClLyuK+W4/TZAH+YBJvPbtRwO2w39iaspvnHBJTKpTNWrx1zi1mBEJ64JIm6ZzaPKr2aGRZckrUeZPMZzNvu54pvPRXT4tEjRsw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1719556006; bh=3Qcu87zHtYudrZVzP+XesbK626KEC2z3x7C5tkvQa+e=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=ZNVaXb5y4UYVS7WuYPs/XS+3dOzgyo1DZvB9GTIpiC3JxyL4P1ZKGeXoYPP9KzIV8eGJIEd++uocP83gYj9oAYw1klDAImUSskLavujOcer6ZboO4V2pl5qQVi042vjsh2BbzElNPSaIKSn4Qr6Zpw2Oq8ecotei+FbQrnYJms5KbUXTNVbhA0b+GmUeOrp7UhFEzab2PIJtWnTShZ9bV5rtI/mkrT+AapsttKAoVaD+4+MMtDFayVOhAXkuJ52pXfYW6XE3F7hUjUu36HZZBBz84xWf953D9Lm53ZJ48ZbRXElNF5+k64hSx7TaciPSWtWKs06t7v+QB0Yd57FVSw== X-YMail-OSG: mV5PbZQVM1kJbiUhRP2DYFB.VzIqvlNsbgnjkRBb_kDMplnFWTf.Rj9l5Awd0C8 Mp4Wq_TE3AH_8d_DWnJ83dr1kv0BOkMY5t2JNY1AdyrCvh8QpMBN6JfbnDR4H_E23RfUhMYXczr1 F4PYPTu677sUps2cWuSqo6JIIYf8jGokS69uFjhGKdNm9rgpQay4p_8DD8.ayZMdWzCHoPR8UBZZ MQjm6DyA4qdBx4qv7.WhhwvD41zexDXYUkKAGZLNtn97kaeBo9_w1QVV8FS.EqQcZUFI3K28Rd7Y ArfFak1nELlRPuAKuoBs2YzdgAnVRXdXNnE9L58DsdmuE324E81WOla4gm57eK5QQH_z7ZbnnD01 lDY6AUYDxIqAMoAsMVjejA.F5FmBwW2xF4g0xAAu9t36cYlMxZuQMpmisrdLz95I1y4Pp0XgEmSy pQoNwZpS54VZtEfrYMTzojhbztov.L4kj9Gfkf7NHA3tdymvfZOZAF_9tR47Zb6RRJAEuL2JCCyv Rz8UaBgHfEGezAaHyBWdTI.imh.jakp9fcCZKeDHWwXLcuiLMI71tLc13SNLssjWd7WYd7DSlmZO KbdH4ce0fvT69ouZxQH7tA2lHwfbt61BXUxsxCxDcPI0VzV7bwV95WB80TtHG9DhleeMcXs_cQVv QzohVz0NF2VtUwEnCQoP4I5B8.K1QONjGsoG0BfQi7sMZQv9JuUJR1ePXrFGtcF.l9SK2GmhxvQd 0NHs.vQ8IOG2.ETMmb24bbIDcwr03dJyx2RzWfj36VOXwfcbVsVIlj08c_W7Jc.LnN51FiHKskj5 DlFTsBXN_SenZp03rzER_Rnj3Ewzmn7.MFY2ataxP_7k_a0I1pr9h4NlCGJteR70Xnnc1aSw4BWy bq3PnUxuPZh1vU8i6QKNkykNFJ0Z3m_E0.nRyzLp_cPtWXAp8D9YewVTpCaQdep0E.Tim6ohDN8W m1cvGkBl2.NS9.kpoIGFG4RKJpKIeqqQLzt76ezIej492R2E9eig3iKD8fEhZrxr8JQiE1Lyz2fZ BOcKyi0RS0x5GkR75qruI_jmj50D5IxaZVcFQ1GQ3sqmMzGNDAF9mxdgVhNjqLY1FZ8jJR71obSQ N2mNowlRmO6WF34fGBThOYhY567Z9kPmYn8DGhMfFm.2MxGcXrs06q31fibgB3Sffgq8SEI0rHNp m2hrEGZ3EHtGSBZtz6HfURzsI3IlQ5DB9x1JThq5oTjWcmaStlAowhwfq09s6EVmNBF4EoWoSnzW 5D2EwM7d9Yrqk6fZ9CuDjyGAHl0nK4gryO5sDsXmxXsSOyehUZWCu6K4D_471Es4Aq67DG4ALrvB ca.a.C8a52j3E.CQ0ffrI0zhcOF3ZReaJ3ygUFh11zl7ar6V13zOUSMSCHQbHZiSKFfJEHKGLOF4 Fp0_1U1Kd53l4ymEHxuG493PC46JXJbMVKDz52jzw9X4DOvIGKzevIBpYilHbs97r2lq26b7sn7M C9XCA4sgDJ1guymerQ9T2juLNhw.UoF9YxPN9FZqU4GXt8F8qguavSZd32xp0xYNeKFpXPXEcwS5 Zmc42iCkc7DEm4rPwVeJ9z7PwyW4trb.yGcPuVfiVaK4WHBtgE3FsUtDRtH0qUodHYnGd.W62KJn N_fha9XiISxKWbaNw70NE8yqypI9IguebbibhjqRlrr70TgOHW4N2Sdl_UEejRiqcw4kQ_Ib3Xgf dSi4iDxTgMEqUK914j8A3ywK4g7peBvXtpBfE9N0f4104KB1Najg7ZZP88e5bUeTGNRTRMx_lQEB Du5aqOJp4Aza.WGbpA.nYOHahC35w.tJHk9dbzDATugbqUlWoQHit9yQIfAcKGJSSuEYJ7__NZ1K GOyNgMVyj2vyfOobqXZCr2KVeYjLakfaKcsk0NvOCWf4pIr5XsI__gIcP0lTobtcQz4a27Qk8D_T eJ23k8nQv0vYsbpE7Hkl1om1AVJQLbzA6SD06268AYgohNq_uEsTgRc6afbjJCZLzvLROc39sRh5 KKgXHnkdIctY5.ccNmU1MZK9GkEYqQg_9hywVr2.7a8r.xOWHoEjc3HE8zYhQ3in4W3rirvxfsVh 6oggXnPFNN1.QgXgpdSIHHNypA2XM0EYdG6w.Hd21cF2Rtik6Idc01HS3iV.GX0ymnvK.VpFzGCu r2L9M19LWjRUFVno6eEPKkMYS0dSeElWRVtrDs8VKIlCzUx.uBWXykBTz9moMp329qdvobznjWL8 G2mZzuy_DZ57GkEcJ6S.qQUNDvhdto8zaC1vovYVKnRcOlGmvUlTvvKAAPsETVdtc.c55j2Oln5u aEKvn7lAIE.z6f0y972UwLoQpySc- X-Sonic-MF: X-Sonic-ID: d7a46355-d9c1-4999-b412-b915dbbadfa4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Fri, 28 Jun 2024 06:26:46 +0000 Received: by hermes--production-gq1-5b4c49485c-sbfr9 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e1cc90feb13c6d2e440f3a0e282ed624; Fri, 28 Jun 2024 06:26:45 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: Git clone failures on armv7, was Re: Git core dump checking out main on armv7 From: Mark Millard In-Reply-To: Date: Thu, 27 Jun 2024 23:26:34 -0700 Cc: FreeBSD ARM List Content-Transfer-Encoding: quoted-printable Message-Id: References: <5D5B6739-1685-43F5-80CC-E55603181D09@yahoo.com> <8F4F4B49-5ED3-4ACA-B0D3-356D8459BE95@yahoo.com> <5DC2D33F-A8DB-4542-B8B3-A131F660A395@yahoo.com> To: Warner Losh , bob prohaska X-Mailer: Apple Mail (2.3774.600.62) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.30:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from] X-Rspamd-Queue-Id: 4W9QSn0JlDz4T2W [Example problems can be reproduced in a context not involving Ethernet or such networking, just system-local activity.] On Jun 27, 2024, at 15:54, Mark Millard wrote: > On Jun 27, 2024, at 14:40, Mark Millard wrote: >=20 >> On Jun 27, 2024, at 12:15, Warner Losh wrote: >>=20 >> On Thu, Jun 27, 2024 at 10:24=E2=80=AFAM bob prohaska = wrote: >>> On Wed, Jun 26, 2024 at 06:24:59PM -0700, Mark Millard wrote: >>>>=20 >>>> Does using the chroot setup using --depth=3D1 on the >>>> RPi2B consistently work when tried repeatedly? Or >>>> was this just an example of a rare success? >>>>=20 >>> Apparently it was a rare success. Five back-to-back retries >>> all failed in an orderly way, though with different messages; >>> invalid fetch-pack and missing blob object. >>> The transcript is at >>> = http://www.zefox.net/~fbsd/rpi2/git_problems/readme_shallow_armv7_chroot_g= ittests >>>=20 >>> Is it worthwhile to repeat with different --depth=3D values? I'm not = sure >>> what the sensible range might be, maybe a 1, 2, 5 sequence? It would >>> be convenient to avoid a panic, as that slows repetition. >>>=20 >>> What happens if you start limiting the memory resources inside a = armv7 jail >>> on a aarch64 machine? >>>=20 >>> Sometimes it works, sometimes it doesn't triggers a "memory = shortage" or >>> "marginal amounts of memory available" bug hunting memories for me. >>=20 >> As I reported in a earlier submittal to the list, I've >> replicated the problem on an armv7 system running main [15] >> with RAM+SWAP being: >>=20 >> 2048 MiBytes RAM + 3685 MiBytes SWAP =3D=3D 5733 MiBytes OVERALL >>=20 >> This was on a Orange Pi+ 2ed. A top variation monitoring and >> reporting various maximum observed figures did not show any >> large memory use compared to even 1024 MiBytes. Any limitation >> would appear to have to be local to some more specific kind >> of constraint rather than overall system RAM or RAM+SWAP. >>=20 >>> Warner I should have noted that, in my context, /tmp is not a tmpfs area. For example: # df -m Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/gpt/BPIM3root 823229 66636 690735 9% / devfs 0 0 0 0% /dev /dev/gpt/BPIM3efi 259 86 173 33% /boot/efi devfs 0 0 0 0% = /usr/obj/DESTDIRs/main-armv7-chroot-ports-official/dev (Also: the activity does not actually involve dev/fd and so it it is not necessary.) > FYI: >=20 > So far, doing the likes of "truss -o ~/truss.txt -f -a -H -p 2136" > towards the end of "Receiving objects" (where 2136 was the original > git process) has always resulted in a normal completion of the clone. >=20 > comparing/contrasting use of >=20 > (gdb) run clone --depth=3D1 -o freebsd = ssh://anongit@192.158.248.9/src.git /tmp/DOES-NOT-EXIST >=20 > Such still gets the errors: >=20 > (gdb) run clone --depth=3D1 -o freebsd = ssh://anongit@192.158.248.9/src.git /tmp/DOES-NOT-EXIST > Starting program: /usr/local/bin/git clone --depth=3D1 -o freebsd = ssh://anongit@192.158.248.9/src.git /tmp/DOES-NOT-EXIST > Cloning into '/tmp/DOES-NOT-EXIST'... > [Detaching after fork from child process 2172] > [New LWP 100254 of process 2171] > [Detaching after fork from child process 2173] > remote: Enumerating objects: 104642, done. > remote: Counting objects: 100% (104642/104642), done. > remote: Compressing objects: 100% (88919/88919), done. > remote: Total 104642 (delta 22161), reused 43523 (delta 11808), = pack-reused 0 (from 0) > Receiving objects: 100% (104642/104642), 344.50 MiB | 1.11 MiB/s, = done. > [LWP 100254 of process 2171 exited] > Resolving deltas: 100% (22161/22161), done. > [Detaching after fork from child process 2176] > fatal: missing blob object '64981a94f867c4c6f9c4aaa26c1117cc8d85de34' > fatal: remote did not send all necessary objects > [Inferior 1 (process 2171) exited with code 0200] Yet another type of test: I'll note that the file:///usr/official-src/ notation used does not automatically involve --local style handling and so is closer to the remote cloning process than --local use would be. Inside a chroot context, with its /usr/official-src/ being via prior mount_nullfs use, the file:///usr/official-src/ notation use case gets example failures: # git clone --depth=3D1 -o freebsd file:///usr/official-src/ = /tmp/DOES-NOT-EXIST Cloning into '/tmp/DOES-NOT-EXIST'... remote: Enumerating objects: 102713, done. remote: Counting objects: 100% (102713/102713), done. remote: Compressing objects: 100% (88060/88060), done. Receiving objects: 100% (102713/102713), 342.13 MiB | 1.10 MiB/s, done. remote: Total 102713 (delta 21887), reused 43790 (delta 10827), = pack-reused 0 (from 0) Resolving deltas: 100% (21887/21887), done. fatal: missing blob object 'b6f146fd872e8006434a846d2536a98b696b9f09' fatal: remote did not send all necessary objects Instead avoiding the mount_nullfs use via duplicating the repository directory tree into the chroot area before doing the chroot did not lead to any failure in my testing. Not involving chroot at all, use of file:///... notation got no failures, independent of direct vs. mount_nullfs use. >>> Thanks for reading, >>>=20 >>> bob prohaska >>>=20 >>>=20 >>>>> A second try without chroot resulted in failure but no panic: >>>>=20 >>>>> : Should own extent_mutex_pool(17) >>>>=20 >>>> That looks like it would be interesting to someone >>>> appropriately knowledgeable. If jemalloc can see bad >>>> mutex ownerships, that seems like such could lead to >>>> all sorts of later problems: Garbage-in/garbage-out. >>>>=20 >>>> I do not know if the message means that various >>>> corruptions might be in place afterwards so that >>>> various later problems might be consequences that >>>> are not surprising possibilities. >>>>=20 >>>>> 47.25 MiB | 1.35 MiB/s =20 >>>>> error: index-pack died of signal 6 >>>>>=20 >>>>> A repeat session produced an oft-seen failure: >>>>>=20 >>>>> root@www:/mnt # mkdir 3rdarmv7gittest >>>>> root@www:/mnt # cd 3rdarmv7gittest >>>>> root@www:/mnt/3rdarmv7gittest # git clone -o freebsd = ssh://anongit@192.158.248.9/src.git . >>>>> Cloning into '.'... >>>>> remote: Enumerating objects: 4511481, done. >>>>> remote: Counting objects: 100% (383480/383480), done. >>>>> remote: Compressing objects: 100% (28955/28955), done. >>>>=20 >>>>> : Should own extent_mutex_pool(17) >>>>=20 >>>> That is the same error notice as above that looked >>>> to be interesting. >>>>=20 >>>> Note that it happens before the later message >>>> "error: index-pack died of signal 6". So that >>>> last may just be a later consequence of the >>>> earlier error(s). >>>>=20 >>>>> 47.25 MiB | 1.35 MiB/s =20 >>>>> error: index-pack died of signal 6 >>>>> fatal: index-pack failed >>>>> root@www:/mnt/3rdarmv7gittest # ls >>>>> root@www:/mnt/3rdarmv7gittest # cd .. >>>>> root@www:/mnt # mkdir 4tharmv7gittest >>>>> root@www:/mnt # cd 4tharmv7gittest >>>>> root@www:/mnt/4tharmv7gittest # git clone -o freebsd = ssh://anongit@192.158.248.9/src.git . >>>>> Cloning into '.'... >>>>> remote: Enumerating objects: 4511481, done. >>>>> remote: Counting objects: 100% (383480/383480), done. >>>>> remote: Compressing objects: 100% (28955/28955), done. >>>>> Receiving objects: 43% (1966916/4511481), 926.00 MiB | 626.00 = KiB/s=20 >>>>> remote: Total 4511481 (delta 377747), reused 354525 (delta = 354525), pack-reused 4128001 (from 1) >>>>> Receiving objects: 100% (4511481/4511481), 1.64 GiB | 705.00 = KiB/s, done. >>>>> fatal: pack is corrupted (SHA1 mismatch) >>>>> fatal: index-pack failed >>>>=20 >>>> Note the lack of a local message: >>>>=20 >>>> : Should own extent_mutex_pool >>>>=20 >>>> But the prior jemalloc message(s) may be sufficient >>>> context to not be surprised about this. >>>>=20 >>>>> root@www:/mnt/4tharmv7gittest #=20 >>>>>=20 >>>>> No panic, however, and it seems reproducible: >>>>> root@www:/mnt # mkdir 5tharmv7gittest >>>>> root@www:/mnt # cd 5tharmv7gittest >>>>> root@www:/mnt/5tharmv7gittest # git clone -o freebsd = ssh://anongit@192.158.248.9/src.git . >>>>> Cloning into '.'... >>>>> remote: Enumerating objects: 4511513, done. >>>>> remote: Counting objects: 100% (383480/383480), done. >>>>> remote: Compressing objects: 100% (28955/28955), done. >>>>> remote: Total 4511513 (delta 377756), reused 354525 (delta = 354525), pack-reused 4128033 (from 1) >>>>> Receiving objects: 100% (4511513/4511513), 1.64 GiB | 1.28 MiB/s, = done. >>>>> fatal: pack is corrupted (SHA1 mismatch) >>>>> fatal: index-pack failed >>>>=20 >>>> Note the lack of a local message: >>>>=20 >>>> : Should own extent_mutex_pool >>>>=20 >>>> But the prior jemalloc message(s) may be sufficient >>>> context to not be surprised about this (again). >>>>=20 >>>>> root@www:/mnt/5tharmv7gittest=20 >>>>>=20 >>>>> Not sure what to try next, thanks for reading this far!=20 >>>>>=20 >>>>> bob prohaska >>>>>=20 >>>>>=20 >>>>> Archived at=20 >>>>> http://www.zefox.net/~fbsd/rpi2/git_problems/readme_armv7 >>>>=20 >>>>=20 >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com