From nobody Thu Jun 27 01:32:15 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W8gzd04vlz5NBtw for ; Thu, 27 Jun 2024 01:32:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4W8gzc0mfMz4qQK for ; Thu, 27 Jun 2024 01:32:28 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-2c8e7553c9eso758713a91.3 for ; Wed, 26 Jun 2024 18:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1719451946; x=1720056746; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=tWMNW83luxEvaoo9PZRBOUTruUwNDd2yTvvFFlIxcMo=; b=Ti/S5oOyyE4rBobhqfZ5m7w75KyVCjBEJhgwZZJwhZVprdEHD0M0mqebG8zGwLUoAg pGmtVTtHcIQMbP1ej4Rc3YiySTKN0GeQQeiWldCTLiEId+i1+d0nhMChEcySMYo7Q7B3 +uwqM+mNfUZjptwvVMEn8xyte+1z0nQpZHCyiwwqDtWkFij9JX2/zVB+sS6IXfa7tKxx Jk50NGKnXUOlxzemBp+GqpXBtljF5FPfy7bXvtkaQK0ZbLIofeRVS2Ua7vJoVwxEZbN6 GoRQU6Gmyp+PKKjUfBVd9DQYyApv4Q77c5Byg7WeMnnJEmGCRkp/EC92+ODhoKgzmLJE qV4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719451946; x=1720056746; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tWMNW83luxEvaoo9PZRBOUTruUwNDd2yTvvFFlIxcMo=; b=KSpJGkzPkDKC+Q9coBBKkqkRCkFg+DMoOQvBDIoTcTHFL4Ij7qKhbOzcdvxj3SOjDs 1yrl0stLK0k9YDzw5cRZVqswGfTe3y2wTdz5VrILHYAywxMv8kNMnjy2YPbzek/MpAtD TdKoKC8+IMgyGSMdWyoLyC9izti7c5srnbHio18Vt1lLe4e+55F9zUGEPLVbHW0pQ6Jw X3Voca9jWOrmFEFeDT4dPo0Hz9eg9EbaVC4RFtn5m0K8aE7McYzCjxBPl+j6wTlGNmPA 8OzDYR59KKT5tpb4vQHnWvHsrNyV8scZhEHp9a74JjgL04FtmrIRF9IvwOuAozhm2Fyd elBA== X-Forwarded-Encrypted: i=1; AJvYcCUCzjU8xGjPFgoNlKLshbUUzx70qqH4RVTy2aQE47IK705QgNJrBrik6UYoppRn4XTOy6ENWQVg63o9ZPWmBcBSbdCxhk/Khg== X-Gm-Message-State: AOJu0Yyf43iP4ShakIjP6zm0Z3WesJK6VzOds1BnKNrtw+ouY2X5iTtU 51jw8ZLXA5Awr0ELJbuSysJHX73Rj3dnitUEwjGKda3BoFECVEA48+KGItXPgEQSP0nB7D7lI+O gbq7gHPTND8Nd/MjCSJipbyWGVYJ4A7+qlAfLgw== X-Google-Smtp-Source: AGHT+IFFU0y3+wlGetSZPlJ7ck6TV8F4KNQyI+lDUOmvR6xtnOP/qsk8ttCy0go3jX2HPtl+N8MMTgDh8TJomebvf+I= X-Received: by 2002:a17:90a:1b87:b0:2bd:9319:3da1 with SMTP id 98e67ed59e1d1-2c86126882amr10453256a91.25.1719451946489; Wed, 26 Jun 2024 18:32:26 -0700 (PDT) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org MIME-Version: 1.0 References: <5D5B6739-1685-43F5-80CC-E55603181D09@yahoo.com> <8F4F4B49-5ED3-4ACA-B0D3-356D8459BE95@yahoo.com> <5DC2D33F-A8DB-4542-B8B3-A131F660A395@yahoo.com> In-Reply-To: <5DC2D33F-A8DB-4542-B8B3-A131F660A395@yahoo.com> From: Warner Losh Date: Wed, 26 Jun 2024 19:32:15 -0600 Message-ID: Subject: Re: Git clone failures on armv7, was Re: Git core dump checking out main on armv7 To: Mark Millard Cc: bob prohaska , FreeBSD ARM List Content-Type: multipart/alternative; boundary="0000000000007abc1d061bd51a40" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4W8gzc0mfMz4qQK --0000000000007abc1d061bd51a40 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jun 26, 2024, 7:25=E2=80=AFPM Mark Millard wrot= e: > On Jun 26, 2024, at 17:42, bob prohaska wrote: > > > On Tue, Jun 25, 2024 at 05:51:59PM -0700, Mark Millard wrote: > >> On Jun 25, 2024, at 17:28, bob prohaska wrote: > >> > >>> On Sun, Jun 23, 2024 at 10:17:53AM -0700, Mark Millard wrote: > >>>> On Jun 23, 2024, at 07:28, bob prohaska wrote: > >>>>>> > >>>>>> It should be possible to mount the media normally used > >>>>>> on one of the RPi2B v1.1's on, say, an RPi4B. One could > >>>>>> then set up enough context to, say, chroot into the > >>>>>> armv7 file system and try executing the clone in that > >>>>>> context. > >>>>>> > >>>>> > >>>>> That's physically not hard to do. Just power down the stable/14 > >>>>> Pi2 and plug the Pi2's boot disk into the Pi4. > >>>> > >>>> Up to power issues, if bus powered. > >>>> > >>>>>> The result would be using the RPi4B aarch64 kernel and > >>>>>> the armv7 world. If that has no problems, then the armv7 > >>>>>> kernel likely has problems that contribute but the armv7 > >>>>>> world would be less likely to. > >>>>>> > >>>>>> Doing this can get into first using the likes of > >>>>>> (notation presumes use of /mnt at the mount point used > >>>>>> above): > >>>>>> > >>>>>> # mount -tdevfs devfs /mnt/dev/ > >>>>>> and possibly: > >>>>>> # mount -tfdescfs none /mnt/fd/ > >>>> > >>>> That last should have been: > >>>> > >>>> # mount -tfdescfs none /mnt/dev/fd/ > >>>> > >>>> Use of the likes of "mkdir -p . . ." may be required in > >>>> order to have the directories for use as mount points > >>>> if they are missing. > >>>> > >>>>>> > >>>>>> before doing the likes of: > >>>>>> > >>>>>> # chroot /mnt/ > >>>>>> # # EXPERIMENT HERE > >>>>>> # exit > >>>>>> > >>>>> I didn't realise that armv7 binaries would run under > >>>>> an aarch64 kernel. Most convenient! > >>>> > >>>> Such can be used to have a faster way to build armv7 > >>>> packages, that also allows larger armv7 processes to > >>>> be involved (but still 32-bit limited with part of > >>>> the address space reserved, for sure). Not only that, > >>>> the process size limit is not a system > >>>> size limit: An RPi4B with 8 GiBytes of RAM can have > >>>> several armv7 processes at once that total over 4 > >>>> GiBytes of RAM in use at the time, no swapping > >>>> needing to be involved. > >>>> > >>>>>> Also, after exiting the chroot session, one would > >>>>>> do the likes of: > >>>>>> > >>>>>> # umount /mnt/dev/ > >>>>>> and possibly: > >>>>>> # umount /mnt/fd/ > >>>> > >>>> That should have been (order dependent for fd inside dev): > >>>> > >>>> possibly: > >>>> # umount /mnt/dev/fd/ > >>>> > >>>> then: > >>>> > >>>> # umount /mnt/dev/ > >>>> > >>>>>> > >>>>>> # umount /mnt/ > >>>>>> > >>>>> Thank you for the cleanup details! > >>>> > >>> > >>> I ended up trying the experiment twice. The first try used the mistak= en > >>> mount -tfdescfs none /mnt/fd/ but since it was noted "possibly" I > ignored > >>> the error and the git clone command completed successfully. After > noticing > >>> your correction, the experiment was repeated and the clone failed: > >>> > >>> root@nemesis:/usr/2ndgittest # git clone --depth=3D1 -o freebsd ssh:/= / > anongit@192.158.248.9/src.git . > >>> Cloning into '.'... > >>> remote: Enumerating objects: 104636, done. > >>> remote: Counting objects: 100% (104636/104636), done. > >>> remote: Compressing objects: 100% (88903/88903), done. > >>> client_loop: send disconnect: Broken pipe88.34 MiB | 165.00 KiB/s > >> > >> The above "client_loop: send disconnect: Broken pipe" > >> seems to be the original error here. > >> > >> It would be interesting to have evidence of how > >> repeatable such is and what variety of other > >> results happen. > > > > Using chroot on the aarch64 -current Pi4 I made > > repeated clones onto the mechanical hard disk > > hosting the armv7 stable/14 filesystem. Couldn't > > reproduce the error above in five tries. > > Transcript at > http://www.zefox.net/~fbsd/rpi2/git_problems/readme_aarch64 > > Sounds like the armv7 kernel likely has some > issues of its own --instead of armv7 world. > Maybe it's just too big? Warner > Then I moved the disk to an armv7 -current Pi2 > > using the same chroot setup, the clone with > > --depth=3D1 worked so I dropped the depth limit > > and repeated; that caused a kernel panic. > > Does using the chroot setup using --depth=3D1 on the > RPi2B consistently work when tried repeatedly? Or > was this just an example of a rare success? > > > A second try without chroot resulted in failure but no panic: > > > : Should own extent_mutex_pool(17) > > That looks like it would be interesting to someone > appropriately knowledgeable. If jemalloc can see bad > mutex ownerships, that seems like such could lead to > all sorts of later problems: Garbage-in/garbage-out. > > I do not know if the message means that various > corruptions might be in place afterwards so that > various later problems might be consequences that > are not surprising possibilities. > > > 47.25 MiB | 1.35 MiB/s > > error: index-pack died of signal 6 > > > > A repeat session produced an oft-seen failure: > > > > root@www:/mnt # mkdir 3rdarmv7gittest > > root@www:/mnt # cd 3rdarmv7gittest > > root@www:/mnt/3rdarmv7gittest # git clone -o freebsd ssh:// > anongit@192.158.248.9/src.git . > > Cloning into '.'... > > remote: Enumerating objects: 4511481, done. > > remote: Counting objects: 100% (383480/383480), done. > > remote: Compressing objects: 100% (28955/28955), done. > > > : Should own extent_mutex_pool(17) > > That is the same error notice as above that looked > to be interesting. > > Note that it happens before the later message > "error: index-pack died of signal 6". So that > last may just be a later consequence of the > earlier error(s). > > > 47.25 MiB | 1.35 MiB/s > > error: index-pack died of signal 6 > > fatal: index-pack failed > > root@www:/mnt/3rdarmv7gittest # ls > > root@www:/mnt/3rdarmv7gittest # cd .. > > root@www:/mnt # mkdir 4tharmv7gittest > > root@www:/mnt # cd 4tharmv7gittest > > root@www:/mnt/4tharmv7gittest # git clone -o freebsd ssh:// > anongit@192.158.248.9/src.git . > > Cloning into '.'... > > remote: Enumerating objects: 4511481, done. > > remote: Counting objects: 100% (383480/383480), done. > > remote: Compressing objects: 100% (28955/28955), done. > > Receiving objects: 43% (1966916/4511481), 926.00 MiB | 626.00 KiB/s > > remote: Total 4511481 (delta 377747), reused 354525 (delta 354525), > pack-reused 4128001 (from 1) > > Receiving objects: 100% (4511481/4511481), 1.64 GiB | 705.00 KiB/s, don= e. > > fatal: pack is corrupted (SHA1 mismatch) > > fatal: index-pack failed > > Note the lack of a local message: > > : Should own extent_mutex_pool > > But the prior jemalloc message(s) may be sufficient > context to not be surprised about this. > > > root@www:/mnt/4tharmv7gittest # > > > > No panic, however, and it seems reproducible: > > root@www:/mnt # mkdir 5tharmv7gittest > > root@www:/mnt # cd 5tharmv7gittest > > root@www:/mnt/5tharmv7gittest # git clone -o freebsd ssh:// > anongit@192.158.248.9/src.git . > > Cloning into '.'... > > remote: Enumerating objects: 4511513, done. > > remote: Counting objects: 100% (383480/383480), done. > > remote: Compressing objects: 100% (28955/28955), done. > > remote: Total 4511513 (delta 377756), reused 354525 (delta 354525), > pack-reused 4128033 (from 1) > > Receiving objects: 100% (4511513/4511513), 1.64 GiB | 1.28 MiB/s, done. > > fatal: pack is corrupted (SHA1 mismatch) > > fatal: index-pack failed > > Note the lack of a local message: > > : Should own extent_mutex_pool > > But the prior jemalloc message(s) may be sufficient > context to not be surprised about this (again). > > > root@www:/mnt/5tharmv7gittest > > > > Not sure what to try next, thanks for reading this far! > > > > bob prohaska > > > > > > Archived at > > http://www.zefox.net/~fbsd/rpi2/git_problems/readme_armv7 > > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > > --0000000000007abc1d061bd51a40 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Jun 26, 2024, 7:25=E2=80=AFPM Mark Millard <= ;marklmi@yahoo.com> wrote:
<= /div>
On Jun 26, 2024, at 17:42, bob prohaska= <fbsd@www.zefox.net> wrote:

> On Tue, Jun 25, 2024 at 05:51:59PM -0700, Mark Millard wrote:
>> On Jun 25, 2024, at 17:28, bob prohaska <fbsd@www.zefox.net= > wrote:
>>
>>> On Sun, Jun 23, 2024 at 10:17:53AM -0700, Mark Millard wrote:<= br> >>>> On Jun 23, 2024, at 07:28, bob prohaska <fbsd@www.zefox= .net> wrote:
>>>>>>
>>>>>> It should be possible to mount the media normally = used
>>>>>> on one of the RPi2B v1.1's on, say, an RPi4B. = One could
>>>>>> then set up enough context to, say, chroot into th= e
>>>>>> armv7 file system and try executing the clone in t= hat
>>>>>> context.
>>>>>>
>>>>>
>>>>> That's physically not hard to do. Just power down = the stable/14
>>>>> Pi2 and plug the Pi2's boot disk into the Pi4.
>>>>
>>>> Up to power issues, if bus powered.
>>>>
>>>>>> The result would be using the RPi4B aarch64 kernel= and
>>>>>> the armv7 world. If that has no problems, then the= armv7
>>>>>> kernel likely has problems that contribute but the= armv7
>>>>>> world would be less likely to.
>>>>>>
>>>>>> Doing this can get into first using the likes of >>>>>> (notation presumes use of /mnt at the mount point = used
>>>>>> above):
>>>>>>
>>>>>> # mount -tdevfs devfs /mnt/dev/
>>>>>> and possibly:
>>>>>> # mount -tfdescfs none /mnt/fd/
>>>>
>>>> That last should have been:
>>>>
>>>> # mount -tfdescfs none /mnt/dev/fd/
>>>>
>>>> Use of the likes of "mkdir -p . . ." may be requ= ired in
>>>> order to have the directories for use as mount points
>>>> if they are missing.
>>>>
>>>>>>
>>>>>> before doing the likes of:
>>>>>>
>>>>>> # chroot /mnt/
>>>>>> # # EXPERIMENT HERE
>>>>>> # exit
>>>>>>
>>>>> I didn't realise that armv7 binaries would run und= er
>>>>> an aarch64 kernel. Most convenient!
>>>>
>>>> Such can be used to have a faster way to build armv7
>>>> packages, that also allows larger armv7 processes to
>>>> be involved (but still 32-bit limited with part of
>>>> the address space reserved, for sure). Not only that,
>>>> the process size limit is not a system
>>>> size limit: An RPi4B with 8 GiBytes of RAM can have
>>>> several armv7 processes at once that total over 4
>>>> GiBytes of RAM in use at the time, no swapping
>>>> needing to be involved.
>>>>
>>>>>> Also, after exiting the chroot session, one would<= br> >>>>>> do the likes of:
>>>>>>
>>>>>> # umount /mnt/dev/
>>>>>> and possibly:
>>>>>> # umount /mnt/fd/
>>>>
>>>> That should have been (order dependent for fd inside dev):=
>>>>
>>>> possibly:
>>>> # umount /mnt/dev/fd/
>>>>
>>>> then:
>>>>
>>>> # umount /mnt/dev/
>>>>
>>>>>>
>>>>>> # umount /mnt/
>>>>>>
>>>>> Thank you for the cleanup details!
>>>>
>>>
>>> I ended up trying the experiment twice. The first try used the= mistaken
>>> mount -tfdescfs none /mnt/fd/ but since it was noted "pos= sibly" I ignored
>>> the error and the git clone command completed successfully. Af= ter noticing
>>> your correction, the experiment was repeated and the clone fai= led:
>>>
>>> root@nemesis:/usr/2ndgittest # git clone --depth=3D1 -o freebs= d ssh://anongit@192.158.248.9/src.git .
>>> Cloning into '.'...
>>> remote: Enumerating objects: 104636, done.
>>> remote: Counting objects: 100% (104636/104636), done.
>>> remote: Compressing objects: 100% (88903/88903), done.
>>> client_loop: send disconnect: Broken pipe88.34 MiB | 165.00 Ki= B/s
>>
>> The above "client_loop: send disconnect: Broken pipe" >> seems to be the original error here.
>>
>> It would be interesting to have evidence of how
>> repeatable such is and what variety of other
>> results happen.
>
> Using chroot on the aarch64 -current Pi4 I made
> repeated clones onto the mechanical hard disk
> hosting the armv7 stable/14 filesystem. Couldn't
> reproduce the error above in five tries.
> Transcript at http://www.= zefox.net/~fbsd/rpi2/git_problems/readme_aarch64

Sounds like the armv7 kernel likely has some
issues of its own --instead of armv7 world.

Maybe it's just too big?

Warner=C2=A0

> Then I moved the disk to an armv7 -current Pi2
> using the same chroot setup, the clone with
> --depth=3D1 worked so I dropped the depth limit
> and repeated; that caused a kernel panic.

Does using the chroot setup using --depth=3D1 on the
RPi2B consistently work when tried repeatedly? Or
was this just an example of a rare success?

> A second try without chroot resulted in failure but no panic:

> <jemalloc>: Should own extent_mutex_pool(17)

That looks like it would be interesting to someone
appropriately knowledgeable. If jemalloc can see bad
mutex ownerships, that seems like such could lead to
all sorts of later problems: Garbage-in/garbage-out.

I do not know if the message means that various
corruptions might be in place afterwards so that
various later problems might be consequences that
are not surprising possibilities.

> 47.25 MiB | 1.35 MiB/s=C2=A0
> error: index-pack died of signal 6
>
> A repeat session produced an oft-seen failure:
>
> root@www:/mnt # mkdir 3rdarmv7gittest
> root@www:/mnt # cd 3rdarmv7gittest
> root@www:/mnt/3rdarmv7gittest # git clone=C2=A0 -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511481, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.

> <jemalloc>: Should own extent_mutex_pool(17)

That is the same error notice as above that looked
to be interesting.

Note that it happens before the later message
"error: index-pack died of signal 6". So that
last may just be a later consequence of the
earlier error(s).

> 47.25 MiB | 1.35 MiB/s=C2=A0
> error: index-pack died of signal 6
> fatal: index-pack failed
> root@www:/mnt/3rdarmv7gittest # ls
> root@www:/mnt/3rdarmv7gittest # cd ..
> root@www:/mnt # mkdir 4tharmv7gittest
> root@www:/mnt # cd 4tharmv7gittest
> root@www:/mnt/4tharmv7gittest # git clone -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511481, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.
> Receiving objects:=C2=A0 43% (1966916/4511481), 926.00 MiB | 626.00 Ki= B/s
> remote: Total 4511481 (delta 377747), reused 354525 (delta 354525), pa= ck-reused 4128001 (from 1)
> Receiving objects: 100% (4511481/4511481), 1.64 GiB | 705.00 KiB/s, do= ne.
> fatal: pack is corrupted (SHA1 mismatch)
> fatal: index-pack failed

Note the lack of a local message:

<jemalloc>: Should own extent_mutex_pool

But the prior jemalloc message(s) may be sufficient
context to not be surprised about this.

> root@www:/mnt/4tharmv7gittest #
>
> No panic, however, and it seems reproducible:
> root@www:/mnt # mkdir 5tharmv7gittest
> root@www:/mnt # cd 5tharmv7gittest
> root@www:/mnt/5tharmv7gittest # git clone -o freebsd ssh://anongit@192.158.248.9/src.git .
> Cloning into '.'...
> remote: Enumerating objects: 4511513, done.
> remote: Counting objects: 100% (383480/383480), done.
> remote: Compressing objects: 100% (28955/28955), done.
> remote: Total 4511513 (delta 377756), reused 354525 (delta 354525), pa= ck-reused 4128033 (from 1)
> Receiving objects: 100% (4511513/4511513), 1.64 GiB | 1.28 MiB/s, done= .
> fatal: pack is corrupted (SHA1 mismatch)
> fatal: index-pack failed

Note the lack of a local message:

<jemalloc>: Should own extent_mutex_pool

But the prior jemalloc message(s) may be sufficient
context to not be surprised about this (again).

> root@www:/mnt/5tharmv7gittest
>
> Not sure what to try next, thanks for reading this far!
>
> bob prohaska
>
>
> Archived at
> http://www.zefox.net/~fbsd/= rpi2/git_problems/readme_armv7



=3D=3D=3D
Mark Millard
marklmi at yahoo.com


--0000000000007abc1d061bd51a40--