From nobody Sun Feb 12 21:25:32 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PFL9r64Vcz3p8gF for ; Sun, 12 Feb 2023 21:25:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PFL9r1PRXz4Ktj for ; Sun, 12 Feb 2023 21:25:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676237150; bh=r+4C5sWthKSxRTkS1yGPu+OmOkMCvRgONJm52BV2gic=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=FcoAGc+6odlgaDDTX01Tj0fQ7pdGoyzsaCBblR0n5Tk8C/5vTQSY1HS/ITd9noYQ2uMnQ/oWQEqDmmk8B+mlRMskCin8bc1Z1sYa0p84hbWGA6zA16vduVAYAb6T4c3IKc4F5h4cl3sT2xksO7gfSRRSfDKRwqBM1z5NW5gP7XtfdHb7iE2GRaholWyLd1ba11kbb4fXscKauBNeGyD/W656we+zwYZ2VB3CYM/sf2/QRHK2ELr9cn3alBtM9i4671++2VnCxkwGlORdVx5XTMZGVX6mY9vhaKeLpFLPi/nnRjSTTp12woHew7A9zzdelHguRdTU/ROduT5YjTIRJg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676237150; bh=HwDVjbu6yjGnogn+wEHLP8JVvx+W9QjXiF4qdm625ZF=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=lYcKBD8EgmCf+DvzEnVzh5NHTNRLKg/EUJ/i4Rmvw7tRlNdDHNF8uUQNYhLzT4UFqcGFeqUMHbQbQtshasxRK4U9a2jbI023AEncFJFr6bf4KNCqpEb2WBsf7aNaL36/rDrLK3n40ZVBNT/NSyK1DLKajtmXXQo4SFGpdSWg7f7N0gdTuu5tE1tZ7BbcGoea7EGf/PVN4NMj0WPwsh1fBllsxPnNj4Ij6HKbRxMxkOei66bUM74MhBbNx9n+HUFayaGHDtsav2zLhDoHppKCh0JDkBgyDfvNxx0GfKaNnBmThHUBCOBp7BXanGC2GvNsBfSmFe8hfOWfBgyEFq5maQ== X-YMail-OSG: ojGQ.dAVM1miy_Rh.OBr2BXeV7h0Oir9QemIetrx41p0d0ba7Njiav5xFGDTF6m X1BM.cXeCMzbJ.MjvdV4OUsmJqadxGlq61HFjXyfElBE5rl8Y43YFaxu0Zxgbi9D_jnFOSt9Zr2A UHh2LA7OVQ_Gp_T4E9Xvg1Ols1Dhh7OdtlCBoIopdBMrG9STC86Q4FyCnDpD0PhOugiAY3mAhuhn WRNcvgrm0qhDANoD0tO59KKlElyZ6URu8gH_nDOy6cyH8P92odJ51wn1tPY2GSqXHTjHvr8TYNVQ kW_bu_dTlT1zeDY3TN7uvAkkT2TCcbqq5ZXiTwOsc_NMJV9bwIVp9efl24dwsZ78_o9pfBzJ.O8j imT_ceKWJSdVlgnEOL11ZWhX81CDiNMvN4FeCDp1Iluu1V9qWZJ47YkRyf3HQLcYsw_3jq7tauLC ePCfdpaHtX6fvrVRj4TwYJoVxGevhuf.FyRZupq1H_eSH8KTSzKB2H374_w8epy4f75A2vgnPvoa LGjPDlgd2Nsc0o.6013WwhedYv21L.Muz_GjFKIJPLH_6ppD04e8K8k7iFj_LgRFEi_.pdsbYBAk i_00bITYMQ2BIzOBHyfWYcy.k5KVD6FTy4V5WwLdVbWV4zaKfIUxGho4ubbCP08QFWAoOvRK46Xk xRzvQEGOcsTN0pWxRAtDlv_8EfhLlcY9.6V8hR0qI8ctgV6TJ0JogHymyJ0ApxPBns0UbAUlHUYQ cL6YZsL09h7mQ94JeXZH7jcLvIeq8UupQKfuCyPt4j8ZNYG0nRyKRjeSgt0mT2zCJ3v4i0pRM.KP Qig5iIC4KMlKJ5d10nTdyWj8Ad6TmznOf_WO2w5NNX2t7pbg4zN.fJ6DQb8GlI1Xzk9kOK.V.zpX S5a65gnUa6nXNXkFZ8OszFqKVadRjG0lDhZPq_k7SuYG.t3hiOUx90zblTntsjB2T5GciWboKpfz Fxr2Ql8UlSj3nI.4jbpjngobJVn0bKjIYMDoKTMFHjlS90Niw3445H176r16KiRr._6inm3Q6UbJ 3vWyuBXuMFp4LYZnTY0vlAtV2VppBc0.x4CpyEVqJWOrbNBeF5xX0aPYm3HCwRNgicpmnJ7MWtDu KzWiyEqindDgqbXR9t219WKJV.yrfD4w9IO2UylXjF_9ETr8Ptph3vLCaMr8EhwYavsSVJyJV5BO BMQcN1jRjuJ9PmIVxXch9Xxcxg9KEsfRq2D8A0WKgN23I0aiolDXi1ALX_OVcfM619v2kTv3ArS_ LSoGq3OdgObLbCK0OFaY5_CL_x_t1.UCbLFvgGiF5GmxF8dBBgZfn_pZTkTgResB.VAoITZB57hG Ig.gUqXdFclatmfTKmZnQnYQ31TkiKoC3I7s_ABL7CETfU76lxZgmyGYQEicQJCccuyjYdzvZPxw rzZaXQcfdcE_PBKrU.8UidxEqETi29c1yeiMOK_Y2SVRWcEbLCaXypfICXrJNIk3OsvdcOjsQ0r_ lnxMsgWzo3zxIcmJX1kUF6BMRY00J6OFVjFuI9A58Bfa1JzmTQLuS549I1BW8UF2ep1Ga5.8R10X _K37PpX.3MpkBtoiCPUqCgWjP34aj2aXxwIWmt6RZwJV9w0nEqRm4RJjsGnsnMUNrTWccZ5IS60d 1DNcK8MsZ6T9eg9DKfXc7NkEv70701dKwerQpBuvpL_rbNhOFT7QPEHx3StaxQBX1JVqYTCwCL45 CWFGFQuumD7pB0p9kHZjOl4XWEHxewORHwPh.wjF_52eWe55.d43sM6cki8tXOs4aqkPvQbsUPGH DfiGLtb7wumC5rBRRx1XlHKubCg.8yCmM47Y55d2tg2kIFPk264RXUmmpA2hPLjyY3JM0W_GPFjd QxLq5Ua8w_310hEd9DdPxRy0nQ_DVnSU.g6.iQRO1UDpRtGhNityIef6eNU.7GQF2rS0FOD2CAPj 1Wpi1NGqrmcGBpdSOxQXwKIK_M74dvNXc4XM.s8CvPk6imD19U_AV0r4CsdETmtNUUfUklWiPNj9 dU2osMBcqHH5HSDYqudDFVhDzO1FmXrwKhQpjcyOPSraK6d1YoZrVGvFkOW476xcg5iXocQlMBob ttZ7TmUQPT4sED1_gomJFsiF55G5BXJPQErbmrHXK8oOrGGpKu2gyR.2IegV9dhbh5HaXdRrXJCK TUUEcTP2MdltlGrMEQlgh9S0_Cuh15JHs1Rjjlk0V1JvigxyBVvdPAjPCipItVQUdj6h0KH.zIlP zMYMw8_MXLlh4r0Zn5jNcyyyrHrScn47Aw28mEMqaMUhfWFlNgb4fBeSdpAYdaASxA4aNvaGYLT_ GRhRtMbAH1mo- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sun, 12 Feb 2023 21:25:50 +0000 Received: by hermes--production-bf1-57c96c66f6-hmvtp (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8863ed9c40677febb79963d3e4810830; Sun, 12 Feb 2023 21:25:44 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: fsck segfaults on rpi3 running 13-stable (and on 14-CURRENT analyzing the same file system that resulted from the 13-STABLE crash) From: Mark Millard In-Reply-To: <20230212195324.GB21535@www.zefox.net> Date: Sun, 12 Feb 2023 13:25:32 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <03840D0B-13D4-4F22-BDAF-2887A4D78BED@yahoo.com> References: <20230211224057.GA17805@www.zefox.net> <9DC74DD9-9AA1-4822-B425-217AAC7DB3F5@yahoo.com> <20230212043524.GA19401@www.zefox.net> <984314A1-FF42-4F92-A212-6BC0D85CB630@yahoo.com> <20230212165333.GB19401@www.zefox.net> <20230212191308.GA21535@www.zefox.net> <20230212195324.GB21535@www.zefox.net> To: bob prohaska , "mckusick@freebsd.org" X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Rspamd-Queue-Id: 4PFL9r1PRXz4Ktj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N [With a backtrace for the fsck_ffs SIGSEGV crash and some listing of code involved, I'm now including mckusick@FreeBSD.org = in the To: . Kirk M. likely would like you to preserve the problematical UFS file system that produces the fsck_ffs crashes, at least for now. For Kirk M.: The below is from/for the fsck_ffs attempted from 14-CURRENT.] On Feb 12, 2023, at 11:53, bob prohaska wrote: > On Sun, Feb 12, 2023 at 11:31:59AM -0800, Mark Millard wrote: >>=20 >> I'll note that another option is to run fsck_ffs from >> lldb in the first place.=20 >=20 > That seems more productive, yielding: >=20 > root@www:~ # lldb /sbin/fsck_ffs > (lldb) target create "/sbin/fsck_ffs" > Current executable set to '/sbin/fsck_ffs' (aarch64). > (lldb) run -fy > Process 62596 launched: '/sbin/fsck_ffs' (aarch64) > usage: fsck_ffs [-BCdEFfnpRrSyZ] [-b block] [-c level] [-m mode] = filesystem ... > Process 62596 exited with status =3D 1 (0x00000001)=20 > (lldb) q > root@www:~ # lldb fsck_ffs > (lldb) target create "fsck_ffs" > Current executable set to 'fsck_ffs' (aarch64). > (lldb) run -fy /dev/da1s2d > Process 62609 launched: '/sbin/fsck_ffs' (aarch64) > ** /dev/da1s2d > ** Last Mounted on /usr > ** Phase 1 - Check Blocks and Sizes > 7912408300994173476 BAD I=3D69393345 > 4313599915630302063 BAD I=3D69393345 > -4473632163892877928 BAD I=3D69393345 > 8068741989830080453 BAD I=3D69393345 > 3857159125896022134 BAD I=3D69393345 > -4354179704011695453 BAD I=3D69393345 > 7611175298055105740 BAD I=3D69393345 > 3985638883347136889 BAD I=3D69393345 > -2495754894521232470 BAD I=3D69393345 > 7739654885841380823 BAD I=3D69393345 > 7912408300994173476 BAD I=3D69393351 > 4313599915630302063 BAD I=3D69393351 > -4473632163892877928 BAD I=3D69393351 > 8068741989830080453 BAD I=3D69393351 > 3857159125896022134 BAD I=3D69393351 > -4354179704011695453 BAD I=3D69393351 > 7611175298055105740 BAD I=3D69393351 > 3985638883347136889 BAD I=3D69393351 > -2495754894521232470 BAD I=3D69393351 > 7739654885841380823 BAD I=3D69393351 > 7912408300994173476 BAD I=3D74682090 > 4313599915630302063 BAD I=3D74682090 > -4473632163892877928 BAD I=3D74682090 > 8068741989830080453 BAD I=3D74682090 > 3857159125896022134 BAD I=3D74682090 > -4354179704011695453 BAD I=3D74682090 > 7611175298055105740 BAD I=3D74682090 > 3985638883347136889 BAD I=3D74682090 > -2495754894521232470 BAD I=3D74682090 > 7739654885841380823 BAD I=3D74682090 > INODE CHECK-HASH FAILED I=3D74999808 OWNER=3D1842251117 MODE=3D15044 > This version of LLDB has no plugin for the language "assembler". = Inspection of frame variables will be limited. > Process 62609 stopped > * thread #1, name =3D 'fsck_ffs', stop reason =3D signal SIGSEGV: = invalid address (fault address: 0x0) > frame #0: 0x00005c6f47c3d550 libc.so.7`strnlen at strnlen.S:50 > 47 bic src, srcin, 15 > 48 mov wtmp, 0xf00f > 49 cbz cntin, L(nomatch) > -> 50 ld1 {vdata.16b}, [src], 16 > 51 dup vrepmask.8h, wtmp > 52 cmeq vhas_chr.16b, vdata.16b, 0 > 53 lsl shift, srcin, 2 > (lldb) bt all > * thread #1, name =3D 'fsck_ffs', stop reason =3D signal SIGSEGV: = invalid address (fault address: 0x0) > * frame #0: 0x00005c6f47c3d550 libc.so.7`strnlen at strnlen.S:50 > frame #1: 0x00005c6f47c08b48 = libc.so.7`__vfprintf(fp=3D0x00005c6f47cd8b68, locale=3D0x00005c6f47cd84a8,= fmt0=3D"MTIME=3D%12.12s %4.4s ", ap=3D(__stack =3D 0x00005c6f45005c40, = __gr_top =3D 0x00005c6f45005bd0, __vr_top =3D 0x00005c6f45005b90, = __gr_offs =3D -48, __vr_offs =3D -128)) at vfprintf.c:854:25 > frame #2: 0x00005c6f47c0752c = libc.so.7`vfprintf_l(fp=3D0x00005c6f47cd8b68, locale=3D0x00005c6f47cd84a8,= fmt0=3D"MTIME=3D%12.12s %4.4s ", ap=3D(__stack =3D 0x00005c6f45005c40, = __gr_top =3D 0x00005c6f45005bd0, __vr_top =3D 0x00005c6f45005b90, = __gr_offs =3D -56, __vr_offs =3D -128)) at vfprintf.c:285:9 > frame #3: 0x00005c6f47c09f94 libc.so.7`vfprintf(fp=3D, = fmt0=3D, ap=3D) at vfprintf.c:292:9 > frame #4: 0x00005c6f47c03dc0 libc.so.7`printf(fmt=3D) = at printf.c:57:8 > frame #5: 0x00005c6ec487edac fsck_ffs`prtinode(ip=3D) = at inode.c:1314:2 > frame #6: 0x00005c6ec487f000 = fsck_ffs`getnextinode(inumber=3D74999808, rebuildcg=3D0) at = inode.c:563:3 > frame #7: 0x00005c6ec4882d5c fsck_ffs`pass1 [inlined] = checkinode(inumber=3D74999808, idesc=3D0x00005c6f45005d20, rebuildcg=3D0) = at pass1.c:254:12 > frame #8: 0x00005c6ec4882d58 fsck_ffs`pass1 at pass1.c:181:8 > frame #9: 0x00005c6ec488209c fsck_ffs`main [inlined] = checkfilesys(filesys=3D) at main.c:446:2 > frame #10: 0x00005c6ec48818b0 fsck_ffs`main(argc=3D1, = argv=3D0x00005c6f45006138) at main.c:210:16 > frame #11: 0x00005c6ec4877ec0 fsck_ffs`__start(argc=3D3, = argv=3D0x00005c6f45006128, env=3D0x00005c6f45006148, = cleanup=3D) at crt1_c.c:72:7 > frame #12: 0x00007813def681d8 ld-elf.so.1`.rtld_start at = rtld_start.S:41 > (lldb) =20 >=20 > Does that make any sense? It gives some context for the internal failure, for sure. I do not see a direct NULL pointer possibility in what I report that I looked at below. It leaves me wondering if something has trashed some memory (stack?) content that is involved. The backtrace indicates a NULL pointer was dereferenced: * thread #1, name =3D 'fsck_ffs', stop reason =3D signal SIGSEGV: = invalid address (fault address: 0x0) * frame #0: 0x00005c6f47c3d550 libc.so.7`strnlen at strnlen.S:50 The "-> 50 ld1 {vdata.16b}, [src], 16" for strnlen indicates that the code is from (given where I have main's source): /usr/main-src/contrib/arm-optimized-routines/string/aarch64/strnlen.S ENTRY (__strnlen_aarch64) PTR_ARG (0) SIZE_ARG (1) bic src, srcin, 15 mov wtmp, 0xf00f cbz cntin, L(nomatch) ld1 {vdata.16b}, [src], 16 . . . This is via the strnlen use in the /usr/main-src/lib/libc/stdio/vfprintf.c code below (leading white space might not be preserved): . . . int __vfprintf(FILE *fp, locale_t locale, const char *fmt0, va_list ap) { . . . case 's': if (flags & LONGINT) { wchar_t *wcp; =20 if (convbuf !=3D NULL) free(convbuf); if ((wcp =3D GETARG(wchar_t *)) =3D=3D = NULL) cp =3D "(null)"; else { convbuf =3D __wcsconv(wcp, = prec); if (convbuf =3D=3D NULL) { fp->_flags |=3D __SERR; goto error; } cp =3D convbuf; } } else if ((cp =3D GETARG(char *)) =3D=3D NULL) cp =3D "(null)"; size =3D (prec >=3D 0) ? strnlen(cp, prec) : = strlen(cp); sign =3D '\0'; break; . . . There are multiple layers involving va_list before we get down to printf in printf.c . I've not tried to validate this va_list related handling. Looking back at code that is inside fsck_ffs source files that leads to the printf usage (and indirectly to the other libc.so code involved) . . . So the code around /usr/main-src/sbin/fsck_ffs/inode.c:1314 looks like: (leading white space might not be preserved) void prtinode(struct inode *ip) { char *p; union dinode *dp; struct passwd *pw; time_t t; dp =3D ip->i_dp; printf(" I=3D%lu ", (u_long)ip->i_number); if (ip->i_number < UFS_ROOTINO || ip->i_number > maxino) return; printf(" OWNER=3D"); if ((pw =3D getpwuid((int)DIP(dp, di_uid))) !=3D NULL) printf("%s ", pw->pw_name); else printf("%u ", (unsigned)DIP(dp, di_uid)); printf("MODE=3D%o\n", DIP(dp, di_mode)); if (preen) printf("%s: ", cdevname); printf("SIZE=3D%ju ", (uintmax_t)DIP(dp, di_size)); t =3D DIP(dp, di_mtime); p =3D ctime(&t); printf("MTIME=3D%12.12s %4.4s ", &p[4], &p[20]); } That, in turned, was called via: ( /usr/main-src/sbin/fsck_ffs/inode.c ) . . . union dinode * getnextinode(ino_t inumber, int rebuildcg) { . . . dp =3D (union dinode *)nextinop; if (sblock.fs_magic =3D=3D FS_UFS1_MAGIC) nextinop +=3D sizeof(struct ufs1_dinode); else nextinop +=3D sizeof(struct ufs2_dinode); if ((ckhashadd & CK_INODE) !=3D 0) { ffs_update_dinode_ckhash(&sblock, (struct ufs2_dinode = *)dp); dirty(&inobuf); } if (ffs_verify_dinode_ckhash(&sblock, (struct ufs2_dinode *)dp) = !=3D 0) { pwarn("INODE CHECK-HASH FAILED"); ip.i_bp =3D NULL; ip.i_dp =3D dp; ip.i_number =3D inumber; prtinode(&ip); if (preen || reply("FIX") !=3D 0) { if (preen) printf(" (FIXED)\n"); ffs_update_dinode_ckhash(&sblock, (struct ufs2_dinode *)dp); dirty(&inobuf); } } . . . In turn: ( /usr/main-src/sbin/fsck_ffs/pass1.c ) . . . static int checkinode(ino_t inumber, struct inodesc *idesc, int rebuildcg) { . . .=20 if ((dp =3D getnextinode(inumber, rebuildcg)) =3D=3D NULL) { pfatal("INVALID INODE"); goto unknown; } . . . In turn (same file): void pass1(void) { . . . /* * Scan the allocated inodes. */ setinodebuf(c, inosused); for (i =3D 0; i < inosused; i++, inumber++) { if (inumber < UFS_ROOTINO) { (void)getnextinode(inumber, rebuildcg); continue; } /* * NULL return indicates probable end of = allocated * inodes during cylinder group rebuild attempt. * We always keep trying until we get to the = minimum * valid number for this cylinder group. */ if (checkinode(inumber, &idesc, rebuildcg) =3D=3D = 0 && i > cgp->cg_initediblk) break; } . . . With that I stop. So far, I've not identified how the NULL pointer showed up that ended up being dereferenced. It does not look likely that I will identify such. =3D=3D=3D Mark Millard marklmi at yahoo.com