From nobody Tue Apr 18 23:09:03 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q1KPC3HsZz45pw4 for ; Tue, 18 Apr 2023 23:09:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q1KPB2xN0z43QD for ; Tue, 18 Apr 2023 23:09:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-94f6c285d22so182137766b.2 for ; Tue, 18 Apr 2023 16:09:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1681859355; x=1684451355; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pWCDpobE6xCBVw1n2TZ2xi4qOsNQRtSaBbrE0kELplQ=; b=zXDzGtl3uH3QvrPEY3+QUS47YanPCSeLsXio7U9cnSVRkp5AidxklnvF+BvLWwIdkb e/XoeR+LRBqidmVwe1xOwMZ0qJrOdNSMaTXpewWsVnQhucvP3Gs1hw4L8YdNr7HHlAG5 JvPGcQVWVbZHzEXGDUkmDyTapXrVIsP+EkDgsKsi+g6JcDXY+lrRc0N7ISbELDQM7VC5 cV/6VvmiSZEJKpEu/8ADW0X4pHo7V41FTQ4e/FA5MQb2UcA9q/DIUqFD7Ju3xcsqeiyn LJStza7jNOrB0gqNZDXIr8kMbJp9KDiat6SynsPRorXNeerh6pFoKxT1srt0eiU9n7DS PwNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681859355; x=1684451355; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pWCDpobE6xCBVw1n2TZ2xi4qOsNQRtSaBbrE0kELplQ=; b=QrTdOLQIMsuxbK0Oj0ejzjElU/MKZaBRuHIWBE+rtfL5tSYrSZ/ir5xqIFPANl8uwv h/HUsFl685dwafBUC6WZdAQnc7JTOlUrT8jSU11/cbz2AdYYuz8ayS8L1NccLUlji7Ys repm0okpSbr+ZCv3wYivliiL9PBbCLP35nvnx7Tk/xEqkGSRoadtVFbAU67lMvzDZmS/ VBj5UnJmp2S2Pe5pKmAt/LI3pBpGw40GHvbeQlE+PLxz9jCkrXcln4m+R/ih9RGPI8+O hXbN1MeQVfMi4Al5UvFP0moBlp4wHtytlLIpWn1ODs0W9ruzmQ6iu5QbyBaWDSs/Knzx oqlQ== X-Gm-Message-State: AAQBX9fYJysew1oQd8dShKA/6Xc1keArS3uZbdNysSY4GHmYY/Hr3nXC 5smSm33Ub/5sXwVXRLNi6AiYz8L12DuPzhBOdoLj9Z4fXtkpp8RA X-Google-Smtp-Source: AKy350b8Wc1KjgR+hvHCSS5WQsiEcMs+6Brvg2xfAeo1OBJSQCjL1Ws5ExLFpaA9A7QG1h4ulHT8IpjDwoTB8uhObAc= X-Received: by 2002:a05:6402:745:b0:501:cced:9c6c with SMTP id p5-20020a056402074500b00501cced9c6cmr3373004edy.7.1681859354849; Tue, 18 Apr 2023 16:09:14 -0700 (PDT) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 References: <6CB8D120-1600-40E6-8A1E-87E709DCEC8F.ref@yahoo.com> <6CB8D120-1600-40E6-8A1E-87E709DCEC8F@yahoo.com> In-Reply-To: From: Warner Losh Date: Tue, 18 Apr 2023 17:09:03 -0600 Message-ID: Subject: Re: The import of openzfs vs. armv7: boot crashs To: Mark Millard Cc: Current FreeBSD , freebsd-arm , Mateusz Guzik , Pawel Jakub Dawidek , Kyle Evans Content-Type: multipart/alternative; boundary="00000000000068890105f9a465db" X-Rspamd-Queue-Id: 4Q1KPB2xN0z43QD X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000068890105f9a465db Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Apr 18, 2023 at 5:04=E2=80=AFPM Mark Millard wr= ote: > On Apr 18, 2023, at 15:46, Warner Losh wrote: > > > Fun... > > > > I'm also fighting aarch64 issues... > > Of what kind? I've been able to use things as committed > in FreeBSD (block_cloning never having been enabled but > jumping from before the import to, effectively, after > the FreeBSD adjustments). But I have not tried anything > that is different as committed in openzfs. > > (I'm one of those that tested poudriere bulk activity > via separate media from my normal aarch64 context. Those > tests had no problems once the full set up adjustments > was present in my context.) > All boot loader issues for special environments. Not in the kernel, so maybe unrelated and I shouldn't have said anything. I'm guessing that upstream needs a generic way to disable all acceleration, even if that has bad performance. I'm looking for a good way to do that (once I get done with the bugs I was fixing when I noticed this issue). Warner > > Warner > > > > On Tue, Apr 18, 2023, 4:45 PM Mark Millard wrote: > > > https://github.com/openzfs/zfs/commit/d0cbd9feaf5b82130f2e679256c71e0c741= 3aae9 > > > > does not seem to cover armv7, just aarch64. (FreeBSD disabled > > floating point for both armv7 and aarch64 but that is a > > different change than above.) > > I probably should have explicitly noted that the fpu disabling > was from after the snapshot being tested here. > > The point of the snapshot test (the most recent available) was > to find out if armv7 crashed before the fpu-use disabling commit. > > > I used: > > > > > FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230406-f21faa67ab6b-262010.img= .xz > > That is from after the import and after: > > =E2=80=A2 git: eb1feadc201a - main - zfs: fix null ap->a_fsizetd NULL= pointer > derefernce Martin Matuska > > but with no other zfs changes. It does not contain: > > =E2=80=A2 git: d6e24901349d - main - zfs: disable kernel fpu usage on= arm and > aarc64 Mateusz Guzik > > (But the openzfs changes are different.) > > > booted an RPi2B v1.1 and tried (note the KSTACK_PAGES notice and the > > "undefined floating point instruction" notice): > > > > # zpool import > > ZFS NOTICE: KSTACK_PAGES is 2 which could result in stack overflow pani= c! > > Please consider adding 'options KSTACK_PAGES=3D4' to your kernel config > > panic: undefined floating point instruction in supervisor mode > > cpuid =3D 2 > > time =3D 1680784610 > > KDB: stack backtrace: > > db_trace_self() at db_trace_self > > pc =3D 0xc05eb154 lr =3D 0xc007a688 (db_trace_self_wrapper+0x= 30) > > sp =3D 0xdd25c480 fp =3D 0xdd25c598 > > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > > pc =3D 0xc007a688 lr =3D 0xc02eb1b4 (vpanic+0x140) > > sp =3D 0xdd25c5a0 fp =3D 0xdd25c5c0 > > r4 =3D 0x00000100 r5 =3D 0x00000000 > > r6 =3D 0xc0736bfc r7 =3D 0xc0b1aea8 > > vpanic() at vpanic+0x140 > > pc =3D 0xc02eb1b4 lr =3D 0xc02eaf94 (doadump) > > sp =3D 0xdd25c5c8 fp =3D 0xdd25c5cc > > r4 =3D 0xc0b92210 r5 =3D 0x00000000 > > r6 =3D 0xc0610ca0 r7 =3D 0xf4210a0d > > r8 =3D 0xddf32e4c r9 =3D 0x00000013 > > r10 =3D 0xdd25c6c0 > > doadump() at doadump > > pc =3D 0xc02eaf94 lr =3D 0xc0610eb0 (vfp_new_thread) > > sp =3D 0xdd25c5d4 fp =3D 0xdd25c638 > > r4 =3D 0xdd25c6c0 r5 =3D 0xdd25c5cc > > r6 =3D 0xc02eaf94 r10 =3D 0xdd25c5d4 > > vfp_new_thread() at vfp_new_thread > > pc =3D 0xc0610eb0 lr =3D 0xc060ff84 (undefinedinstruction+0x1= 78) > > sp =3D 0xdd25c640 fp =3D 0xdd25c6b8 > > undefinedinstruction() at undefinedinstruction+0x178 > > pc =3D 0xc060ff84 lr =3D 0xc05edaa8 (exception_exit) > > sp =3D 0xdd25c6c0 fp =3D 0xdd25c750 > > r4 =3D 0x20000013 r5 =3D 0xde45e000 > > r6 =3D 0xdd25c890 r7 =3D 0xdd25c8b0 > > r8 =3D 0x00000000 r9 =3D 0x00000000 > > r10 =3D 0xdd25c8c0 > > exception_exit() at exception_exit > > pc =3D 0xc05edaa8 lr =3D 0xddf31f20 (K256) > > sp =3D 0xdd25c750 fp =3D 0xdd25c750 > > r0 =3D 0xdd25c890 r1 =3D 0xde45e000 > > r2 =3D 0xde45e400 r3 =3D 0xddf309fc > > r4 =3D 0x00000400 r5 =3D 0xde45e000 > > r6 =3D 0xdd25c890 r7 =3D 0xdd25c8b0 > > r8 =3D 0x00000000 r9 =3D 0x00000000 > > r10 =3D 0xdd25c8c0 r12 =3D 0xdd25c7a0 > > zfs_sha256_block_neon() at zfs_sha256_block_neon+0x1c > > pc =3D 0xddf32e4c lr =3D 0xc0946e8c (pcpup) > > sp =3D 0xdd25c758 fp =3D 0xc0b0aeec > > r4 =3D 0xc0919610 r5 =3D 0xc0919630 > > r6 =3D 0xc0919618 r7 =3D 0x642ebce2 > > r8 =3D 0xc0b1b0ec r9 =3D 0xc0915e88 > > r10 =3D 0xc0b1b0dc > > Fatal kernel mode data abort: 'Translation Fault (L1)' on read > > trapframe: 0xdd25c330 > > FSR=3D00000005, FAR=3D95e29398, spsr=3D200000d3 > > r0 =3Ddd25c424, r1 =3D81000000, r2 =3D95e29395, r3 =3D55555555 > > r4 =3Dc08ae93c, r5 =3D00004aa0, r6 =3D00004aa0, r7 =3Dc08d3e3c > > r8 =3D00000001, r9 =3Dc079567a, r10=3D0000000b, r11=3Ddd25c3e0 > > r12=3D00000000, ssp=3Ddd25c3c4, slr=3D00000001, pc =3Dc0610308 > > > > panic: Fatal abort > > . . . (repeats over and over) . . . > > > > > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --00000000000068890105f9a465db Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Apr 18, 2023 at 5:04=E2=80=AF= PM Mark Millard <marklmi@yahoo.com<= /a>> wrote:
O= n Apr 18, 2023, at 15:46, Warner Losh <imp@bsdimp.com> wrote:

> Fun...
>
> I'm also fighting aarch64 issues...

Of what kind? I've been able to use things as committed
in FreeBSD (block_cloning never having been enabled but
jumping from before the import to, effectively, after
the FreeBSD adjustments). But I have not tried anything
that is different as committed in openzfs.

(I'm one of those that tested poudriere bulk activity
via separate media from my normal aarch64 context. Those
tests had no problems once the full set up adjustments
was present in my context.)

All boot lo= ader issues for special environments. Not in the kernel,
so maybe= unrelated and I shouldn't have said anything.

I'm guessing that upstream needs a generic way to disable all
acceleration, even if that has bad performance. I'm looking for a
good way to do that (once I get done with the bugs I was fixing
when I noticed this issue).

Warner
=C2=A0
> Warner
>
> On Tue, Apr 18, 2023, 4:45 PM Mark Millard <marklmi@yahoo.com> wrote:
> https://github.co= m/openzfs/zfs/commit/d0cbd9feaf5b82130f2e679256c71e0c7413aae9
>
> does not seem to cover armv7, just aarch64. (FreeBSD disabled
> floating point for both armv7 and aarch64 but that is a
> different change than above.)

I probably should have explicitly noted that the fpu disabling
was from after the snapshot being tested here.

The point of the snapshot test (the most recent available) was
to find out if armv7 crashed before the fpu-use disabling commit.

> I used:
>
> FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230406-f21faa67ab6b-262010.= img.xz

That is from after the import and after:

=C2=A0 =C2=A0 =E2=80=A2 git: eb1feadc201a - main - zfs: fix null ap->a_f= sizetd NULL pointer derefernce Martin Matuska

but with no other zfs changes. It does not contain:

=C2=A0 =C2=A0 =E2=80=A2 git: d6e24901349d - main - zfs: disable kernel fpu = usage on arm and aarc64 Mateusz Guzik

(But the openzfs changes are different.)

> booted an RPi2B v1.1 and tried (note the KSTACK_PAGES notice and the > "undefined floating point instruction" notice):
>
> # zpool import
> ZFS NOTICE: KSTACK_PAGES is 2 which could result in stack overflow pan= ic!
> Please consider adding 'options KSTACK_PAGES=3D4' to your kern= el config
> panic: undefined floating point instruction in supervisor mode
> cpuid =3D 2
> time =3D 1680784610
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc05eb154=C2=A0 lr =3D 0xc00= 7a688 (db_trace_self_wrapper+0x30)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c480=C2=A0 fp =3D 0xdd2= 5c598
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc007a688=C2=A0 lr =3D 0xc02= eb1b4 (vpanic+0x140)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c5a0=C2=A0 fp =3D 0xdd2= 5c5c0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0x00000100=C2=A0 r5 =3D 0x000= 00000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xc0736bfc=C2=A0 r7 =3D 0xc0b= 1aea8
> vpanic() at vpanic+0x140
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc02eb1b4=C2=A0 lr =3D 0xc02= eaf94 (doadump)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c5c8=C2=A0 fp =3D 0xdd2= 5c5cc
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0xc0b92210=C2=A0 r5 =3D 0x000= 00000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xc0610ca0=C2=A0 r7 =3D 0xf42= 10a0d
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r8 =3D 0xddf32e4c=C2=A0 r9 =3D 0x000= 00013
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0r10 =3D 0xdd25c6c0
> doadump() at doadump
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc02eaf94=C2=A0 lr =3D 0xc06= 10eb0 (vfp_new_thread)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c5d4=C2=A0 fp =3D 0xdd2= 5c638
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0xdd25c6c0=C2=A0 r5 =3D 0xdd2= 5c5cc
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xc02eaf94 r10 =3D 0xdd25c5d4=
> vfp_new_thread() at vfp_new_thread
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc0610eb0=C2=A0 lr =3D 0xc06= 0ff84 (undefinedinstruction+0x178)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c640=C2=A0 fp =3D 0xdd2= 5c6b8
> undefinedinstruction() at undefinedinstruction+0x178
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc060ff84=C2=A0 lr =3D 0xc05= edaa8 (exception_exit)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c6c0=C2=A0 fp =3D 0xdd2= 5c750
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0x20000013=C2=A0 r5 =3D 0xde4= 5e000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xdd25c890=C2=A0 r7 =3D 0xdd2= 5c8b0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r8 =3D 0x00000000=C2=A0 r9 =3D 0x000= 00000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0r10 =3D 0xdd25c8c0
> exception_exit() at exception_exit
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xc05edaa8=C2=A0 lr =3D 0xddf= 31f20 (K256)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c750=C2=A0 fp =3D 0xdd2= 5c750
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r0 =3D 0xdd25c890=C2=A0 r1 =3D 0xde4= 5e000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r2 =3D 0xde45e400=C2=A0 r3 =3D 0xddf= 309fc
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0x00000400=C2=A0 r5 =3D 0xde4= 5e000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xdd25c890=C2=A0 r7 =3D 0xdd2= 5c8b0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r8 =3D 0x00000000=C2=A0 r9 =3D 0x000= 00000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0r10 =3D 0xdd25c8c0 r12 =3D 0xdd25c7a0=
> zfs_sha256_block_neon() at zfs_sha256_block_neon+0x1c
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pc =3D 0xddf32e4c=C2=A0 lr =3D 0xc09= 46e8c (pcpup)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sp =3D 0xdd25c758=C2=A0 fp =3D 0xc0b= 0aeec
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r4 =3D 0xc0919610=C2=A0 r5 =3D 0xc09= 19630
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r6 =3D 0xc0919618=C2=A0 r7 =3D 0x642= ebce2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 r8 =3D 0xc0b1b0ec=C2=A0 r9 =3D 0xc09= 15e88
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0r10 =3D 0xc0b1b0dc
> Fatal kernel mode data abort: 'Translation Fault (L1)' on read=
> trapframe: 0xdd25c330
> FSR=3D00000005, FAR=3D95e29398, spsr=3D200000d3
> r0 =3Ddd25c424, r1 =3D81000000, r2 =3D95e29395, r3 =3D55555555
> r4 =3Dc08ae93c, r5 =3D00004aa0, r6 =3D00004aa0, r7 =3Dc08d3e3c
> r8 =3D00000001, r9 =3Dc079567a, r10=3D0000000b, r11=3Ddd25c3e0
> r12=3D00000000, ssp=3Ddd25c3c4, slr=3D00000001, pc =3Dc0610308
>
> panic: Fatal abort
> . . . (repeats over and over) . . .
>




=3D=3D=3D
Mark Millard
marklmi at yahoo.com

--00000000000068890105f9a465db--