Re: Git clone failures on armv7, was Re: Git core dump checking out main on armv7

On Jun 26, 2024, at 17:42, bob prohaska <fbsd@www.zefox.net> wrote:

> On Tue, Jun 25, 2024 at 05:51:59PM -0700, Mark Millard wrote:
>> On Jun 25, 2024, at 17:28, bob prohaska <fbsd@www.zefox.net> wrote:
>> 
>>> On Sun, Jun 23, 2024 at 10:17:53AM -0700, Mark Millard wrote:
>>>> On Jun 23, 2024, at 07:28, bob prohaska <fbsd@www.zefox.net> wrote:
>>>>>> 
>>>>>> It should be possible to mount the media normally used
>>>>>> on one of the RPi2B v1.1's on, say, an RPi4B. One could
>>>>>> then set up enough context to, say, chroot into the
>>>>>> armv7 file system and try executing the clone in that
>>>>>> context.
>>>>>> 
>>>>> 
>>>>> That's physically not hard to do. Just power down the stable/14
>>>>> Pi2 and plug the Pi2's boot disk into the Pi4.
>>>> 
>>>> Up to power issues, if bus powered.
>>>> 
>>>>>> The result would be using the RPi4B aarch64 kernel and
>>>>>> the armv7 world. If that has no problems, then the armv7
>>>>>> kernel likely has problems that contribute but the armv7
>>>>>> world would be less likely to.
>>>>>> 
>>>>>> Doing this can get into first using the likes of
>>>>>> (notation presumes use of /mnt at the mount point used
>>>>>> above):
>>>>>> 
>>>>>> # mount -tdevfs devfs /mnt/dev/
>>>>>> and possibly:
>>>>>> # mount -tfdescfs none /mnt/fd/
>>>> 
>>>> That last should have been:
>>>> 
>>>> # mount -tfdescfs none /mnt/dev/fd/
>>>> 
>>>> Use of the likes of "mkdir -p . . ." may be required in
>>>> order to have the directories for use as mount points
>>>> if they are missing.
>>>> 
>>>>>> 
>>>>>> before doing the likes of:
>>>>>> 
>>>>>> # chroot /mnt/
>>>>>> # # EXPERIMENT HERE
>>>>>> # exit
>>>>>> 
>>>>> I didn't realise that armv7 binaries would run under
>>>>> an aarch64 kernel. Most convenient!
>>>> 
>>>> Such can be used to have a faster way to build armv7
>>>> packages, that also allows larger armv7 processes to
>>>> be involved (but still 32-bit limited with part of
>>>> the address space reserved, for sure). Not only that,
>>>> the process size limit is not a system
>>>> size limit: An RPi4B with 8 GiBytes of RAM can have
>>>> several armv7 processes at once that total over 4
>>>> GiBytes of RAM in use at the time, no swapping
>>>> needing to be involved.
>>>> 
>>>>>> Also, after exiting the chroot session, one would
>>>>>> do the likes of:
>>>>>> 
>>>>>> # umount /mnt/dev/
>>>>>> and possibly:
>>>>>> # umount /mnt/fd/
>>>> 
>>>> That should have been (order dependent for fd inside dev):
>>>> 
>>>> possibly:
>>>> # umount /mnt/dev/fd/
>>>> 
>>>> then:
>>>> 
>>>> # umount /mnt/dev/
>>>> 
>>>>>> 
>>>>>> # umount /mnt/
>>>>>> 
>>>>> Thank you for the cleanup details!
>>>> 
>>> 
>>> I ended up trying the experiment twice. The first try used the mistaken
>>> mount -tfdescfs none /mnt/fd/ but since it was noted "possibly" I ignored
>>> the error and the git clone command completed successfully. After noticing
>>> your correction, the experiment was repeated and the clone failed:
>>> 
>>> root@nemesis:/usr/2ndgittest # git clone --depth=1 -o freebsd ssh://anongit@192.158.248.9/src.git .
>>> Cloning into '.'...
>>> remote: Enumerating objects: 104636, done.
>>> remote: Counting objects: 100% (104636/104636), done.
>>> remote: Compressing objects: 100% (88903/88903), done.
>>> client_loop: send disconnect: Broken pipe88.34 MiB | 165.00 KiB/s
>> 
>> The above "client_loop: send disconnect: Broken pipe"
>> seems to be the original error here.
>> 
>> It would be interesting to have evidence of how
>> repeatable such is and what variety of other
>> results happen.
> 
> Using chroot on the aarch64 -current Pi4 I made
> repeated clones onto the mechanical hard disk
> hosting the armv7 stable/14 filesystem. Couldn't
> reproduce the error above in five tries.
> Transcript at http://www.zefox.net/~fbsd/rpi2/git_problems/readme_aarch64

Sounds like the armv7 kernel likely has some
issues of its own --instead of armv7 world.

> Then I moved the disk to an armv7 -current Pi2
> using the same chroot setup, the clone with 
> --depth=1 worked so I dropped the depth limit
> and repeated; that caused a kernel panic.

Does using the chroot setup using --depth=1 on the
RPi2B consistently work when tried repeatedly? Or
was this just an example of a rare success?

> A second try without chroot resulted in failure but no panic:

> <jemalloc>: Should own extent_mutex_pool(17)

That looks like it would be interesting to someone
appropriately knowledgeable. If jemalloc can see bad
mutex ownerships, that seems like such could lead to
all sorts of later problems: Garbage-in/garbage-out.

I do not know if the message means that various
corruptions might be in place afterwards so that
various later problems might be consequences that
are not surprising possibilities.

> 47.25 MiB | 1.35 MiB/s  
> error: index-pack died of signal 6
> 
> A repeat session produced an oft-seen failure:
> 
> root@www:/mnt # mkdir 3rdarmv7gittest
> root@www:/mnt # cd 3rdarmv7gittest
> root@www:/mnt/3rdarmv7gittest # git clone  -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511481, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.

> <jemalloc>: Should own extent_mutex_pool(17)

That is the same error notice as above that looked
to be interesting.

Note that it happens before the later message
"error: index-pack died of signal 6". So that
last may just be a later consequence of the
earlier error(s).

> 47.25 MiB | 1.35 MiB/s  
> error: index-pack died of signal 6
> fatal: index-pack failed
> root@www:/mnt/3rdarmv7gittest # ls
> root@www:/mnt/3rdarmv7gittest # cd ..
> root@www:/mnt # mkdir 4tharmv7gittest
> root@www:/mnt # cd 4tharmv7gittest
> root@www:/mnt/4tharmv7gittest # git clone -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511481, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.
> Receiving objects:  43% (1966916/4511481), 926.00 MiB | 626.00 KiB/s 
> remote: Total 4511481 (delta 377747), reused 354525 (delta 354525), pack-reused 4128001 (from 1)
> Receiving objects: 100% (4511481/4511481), 1.64 GiB | 705.00 KiB/s, done.
> fatal: pack is corrupted (SHA1 mismatch)
> fatal: index-pack failed

Note the lack of a local message:

<jemalloc>: Should own extent_mutex_pool

But the prior jemalloc message(s) may be sufficient
context to not be surprised about this.

> root@www:/mnt/4tharmv7gittest # 
> 
> No panic, however, and it seems reproducible:
> root@www:/mnt # mkdir 5tharmv7gittest
> root@www:/mnt # cd 5tharmv7gittest
> root@www:/mnt/5tharmv7gittest # git clone -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511513, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.
> remote: Total 4511513 (delta 377756), reused 354525 (delta 354525), pack-reused 4128033 (from 1)
> Receiving objects: 100% (4511513/4511513), 1.64 GiB | 1.28 MiB/s, done.
> fatal: pack is corrupted (SHA1 mismatch)
> fatal: index-pack failed

Note the lack of a local message:

<jemalloc>: Should own extent_mutex_pool

But the prior jemalloc message(s) may be sufficient
context to not be surprised about this (again).

> root@www:/mnt/5tharmv7gittest 
> 
> Not sure what to try next, thanks for reading this far! 
> 
> bob prohaska
> 
> 
> Archived at 
> http://www.zefox.net/~fbsd/rpi2/git_problems/readme_armv7



===
Mark Millard
marklmi at yahoo.com