From nobody Mon Jul 17 01:17:09 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R441z5S7cz4nHnx for ; Mon, 17 Jul 2023 01:17:27 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-yb1-xb2b.google.com (mail-yb1-xb2b.google.com [IPv6:2607:f8b0:4864:20::b2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R441z3kFzz43tq for ; Mon, 17 Jul 2023 01:17:27 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb2b.google.com with SMTP id 3f1490d57ef6-bad0c4f6f50so5687057276.1 for ; Sun, 16 Jul 2023 18:17:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689556646; x=1692148646; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=W6SZRIwBCvUdhNVj4tfegkNuK0u5O1RAENCWDrQJO4w=; b=BlvDH8R2GnkNE3oWq8CXnOs9WDGn4HLR3JCOYVg3M5fJVAA4TDIa7HHiSnVKp1Zbus cR0plSM/h8BoOFsQHTCNNikAUX//AVGXE6aHBtIkSxXWyM81Gvnt0adrcFKqCVr2q56I kOmbEfo4I2Ub9GD2ISEAbHJw+tR7ujPWEB1DJzhdS/p/l5/aAJiwRR+kfLMWIP33Gz02 QEQB7J5xsTBgiq6bpabnLGhW11MoiLCSLOmPp1perGtXisKaxT/hVNhEA0QOcNBLwNI5 5C58nXl0Ut5YW7EIhg/7Sd3ZFN+lM59huTb2IS5/9RygJ48nWid7P+quGRb3NHzypGQm WTVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689556646; x=1692148646; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=W6SZRIwBCvUdhNVj4tfegkNuK0u5O1RAENCWDrQJO4w=; b=Wb3kjnnKtUoZ25S5uzjNN0YKRsTaxq0HoYKhqnYaNDkPlT98Nu5I1RrU++DxalYB24 04SUKKPGQ77fu+Ve+d8C0JTYwbvfyCF6wqVPRl0vsk/drRXraiiq6JOOB1JnDySu2vQN CnUQce7omrn3NG2v2QeQ7lGmNxkO2UQzd5CmZsPVqfVhZiWimEL0YXQo8nTlRyil4ydM jbzlgIALF5sVKaSJYGQzDtA7vGzo5aBUbYi9Yv5g/tWrNJnK1mjXRyFitV65JeKCxeRY Qwx0tOk+3iampvrV86Ihz/+4Y02r3pm2coVbtkLEYoBE2fbTZf+fCRNVJALNmrOeEUw+ Yysw== X-Gm-Message-State: ABy/qLZqbARaCiVLPip4O1tCHLn6FAA+Qw/J6liraw3FfmaY4lTuijpP w6zqS/Hb5DSNgsHTXRxXVNmVf8nXQH3loel+2V18BZG1S78= X-Google-Smtp-Source: APBJJlHKw93Nn1i7tc5mv3bXuUVTnQA/0wiwruZbZI4d/oIK3CdkjDAbtsCgaHgeVt/BJzvqOvYVQNNZeqUDTmgOZAA= X-Received: by 2002:a25:a184:0:b0:cb8:a812:a91 with SMTP id a4-20020a25a184000000b00cb8a8120a91mr8596480ybi.0.1689556646080; Sun, 16 Jul 2023 18:17:26 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <485EB974-5EF4-4569-9B40-81A474983F33.ref@yahoo.com> <485EB974-5EF4-4569-9B40-81A474983F33@yahoo.com> In-Reply-To: <485EB974-5EF4-4569-9B40-81A474983F33@yahoo.com> From: Kevin Oberman Date: Sun, 16 Jul 2023 18:17:09 -0700 Message-ID: Subject: Re: rsync use with -tmsdosfs mounted file system? file has vanished: . . . To: Mark Millard Cc: Current FreeBSD Content-Type: multipart/alternative; boundary="000000000000b7b2ec0600a48f90" X-Rspamd-Queue-Id: 4R441z3kFzz43tq X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000b7b2ec0600a48f90 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jul 16, 2023 at 1:57=E2=80=AFPM Mark Millard wr= ote: > I used a sequence that looked like: > > mount -onoatime -tmsdosfs /dev/gpt/CA72optM2efi /CA72optM2efi-media/ \ > && rsync -x --delete -aAUHhh --info=3Dprogress2 /boot/efi/ > /CA72optM2efi-media/ > > that got: > > file has vanished: "/CA72optM2efi-media/BCM271~5.DTB" > file has vanished: "/CA72optM2efi-media/BCM271~6.DTB" > 73.77K 0% 1.63MB/s 0:00:00 (xfr#7, to-chk=3D0/493) rsy= nc > warning: some files vanished before they could be transferred (code 24) a= t > main.c(1357) [sender=3D3.2.7] > > After that, activity reported the likes of: > > rsync: [generator] delete_file: unlink(overlays/VC4-KM~8.DTB) failed: > Read-only file system (30) > and: > rsync: [receiver] mkstemp "/CA72optM2efi-media/.fixup4.dat.2Wonu9" failed= : > Read-only file system (30) > > More than rsync was odd at that point: > > # ls -Tld /CA72optM2efi-media/*.DTB > ls: /CA72optM2efi-media/BCM271~5.DTB: No such file or directory > ls: /CA72optM2efi-media/BCM271~6.DTB: No such file or directory > > # rm /CA72optM2efi-media/*/*.DTB > override rwxr-xr-x root/wheel uarch for > /CA72optM2efi-media/overlays/SDHOST~1.DTB? y > rm: /CA72optM2efi-media/overlays/SDHOST~1.DTB: Read-only file system > . . . > > But: > > # mount | grep media > /dev/gpt/CA72optM2efi on /CA72optM2efi-media (msdosfs, local, noatime) > > So the mount itself was not the source of the read-only status so far. > > I then tried: > > # umount /CA72optM2efi-media > # newfs_msdos /dev/da0p1 > # mount -onoatime -tmsdosfs /dev/gpt/CA72optM2efi /mnt > # cp -aRx /boot/efi/ /mnt/ > cp: utimensat: /mnt: Invalid argument > > (which is normal). > > # umount /mnt > > No more oddities , so far after that. > > > For reference: > > # uname -apKU > FreeBSD CA72-16Gp-ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #99 > main-n264171-2a0c0aea4209-dirty: Fri Jul 14 21:00:44 PDT 2023 > root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/ar= m64.aarch64/sys/GENERIC-NODBG-CA72 > arm64 aarch64 1400093 1400093 > > # pkg info rsync > rsync-3.2.7 > Name : rsync > Version : 3.2.7 > Installed on : Sat Jul 15 14:53:48 2023 PDT > Origin : net/rsync > Architecture : FreeBSD:14:aarch64 > . . . > Annotations : > FreeBSD_version: 1400092 > build_timestamp: 2023-07-02T06:57:44+0000 > built_by : poudriere-git-3.3.99.20220831 > cpe : cpe:2.3:a:samba:rsync:3.2.7:::::freebsd14:aarch64 > port_checkout_unclean: no > port_git_hash : f45cd5bd9d4b > ports_top_checkout_unclean: yes > ports_top_git_hash: 880f72e54deb > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.co This looks a bit like an issue I was hitting on a 4 CPU, 4 thread Alder Lake processor and 500GB SSD running 13.2-RELEASE. I saw several very strange corruptions, at least one "rsync warning: some files vanished before they could be transferred". In one (the last) case, the system crashed. The 'disc' was corrupted badly enough that fsck failed and I could not boot up the system. The disk was UFS2 GPT format and EFI boot. The interface is SATA, not nVME. In this case, I was installing on a new system and copying the majority of the file system from my old system. The 'fix' is strange and probably not one many other can use. I installed a spinning rust drive of 500GB and installed FreeBSD and used rsync again and it worked. I can't say whether it was a fluke that it worked, but it really smells like some sort of race condition. Could be rsync , VFS, or device driver. Since then I have seen one crash while backing up the system disk using rsync. No corruption and doing another rsync after reboot worked fine, but it was a much smaller run as the first attempt was nearly complete when the system crashed. Maybe unrelated. I do have the core file from the crash. Stil, something weird has been going on. Same issue on two identical systems, so not likely hardware. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000b7b2ec0600a48f90 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sun, Jul 16, 2023 at 1:57=E2= =80=AFPM Mark Millard <marklmi@yaho= o.com> wrote:
I used a sequence that looked like:

mount -onoatime -tmsdosfs /dev/gpt/CA72optM2efi /CA72optM2efi-media/ \
&& rsync -x --delete -aAUHhh --info=3Dprogress2 /boot/efi/ /CA72opt= M2efi-media/

that got:

file has vanished: "/CA72optM2efi-media/BCM271~5.DTB"
file has vanished: "/CA72optM2efi-media/BCM271~6.DTB"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A073.77K=C2=A0 =C2=A00%=C2=A0 =C2=A0 1.63MB= /s=C2=A0 =C2=A0 0:00:00 (xfr#7, to-chk=3D0/493)=C2=A0 =C2=A0rsync warning: = some files vanished before they could be transferred (code 24) at main.c(13= 57) [sender=3D3.2.7]

After that, activity reported the likes of:

rsync: [generator] delete_file: unlink(overlays/VC4-KM~8.DTB) failed: Read-= only file system (30)
and:
rsync: [receiver] mkstemp "/CA72optM2efi-media/.fixup4.dat.2Wonu9"= ; failed: Read-only file system (30)

More than rsync was odd at that point:

# ls -Tld /CA72optM2efi-media/*.DTB
ls: /CA72optM2efi-media/BCM271~5.DTB: No such file or directory
ls: /CA72optM2efi-media/BCM271~6.DTB: No such file or directory

# rm /CA72optM2efi-media/*/*.DTB
override rwxr-xr-x root/wheel uarch for /CA72optM2efi-media/overlays/SDHOST= ~1.DTB? y
rm: /CA72optM2efi-media/overlays/SDHOST~1.DTB: Read-only file system
. . .

But:

# mount | grep media
/dev/gpt/CA72optM2efi on /CA72optM2efi-media (msdosfs, local, noatime)

So the mount itself was not the source of the read-only status so far.

I then tried:

# umount /CA72optM2efi-media
# newfs_msdos /dev/da0p1
# mount -onoatime -tmsdosfs /dev/gpt/CA72optM2efi /mnt
# cp -aRx /boot/efi/ /mnt/
cp: utimensat: /mnt: Invalid argument

(which is normal).

# umount /mnt

No more oddities , so far after that.


For reference:

# uname -apKU
FreeBSD CA72-16Gp-ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #99 main-n264171-2a= 0c0aea4209-dirty: Fri Jul 14 21:00:44 PDT 2023=C2=A0 =C2=A0 =C2=A0root@CA72= -16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/= sys/GENERIC-NODBG-CA72 arm64 aarch64 1400093 1400093

# pkg info rsync
rsync-3.2.7
Name=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: rsync
Version=C2=A0 =C2=A0 =C2=A0 =C2=A0 : 3.2.7
Installed on=C2=A0 =C2=A0: Sat Jul 15 14:53:48 2023 PDT
Origin=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: net/rsync
Architecture=C2=A0 =C2=A0: FreeBSD:14:aarch64
. . .
Annotations=C2=A0 =C2=A0 :
FreeBSD_version: 1400092
build_timestamp: 2023-07-02T06:57:44+0000
built_by=C2=A0 =C2=A0 =C2=A0 =C2=A0: poudriere-git-3.3.99.20220831
cpe=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : cpe:2.3:a:samba:rsync:3.2.7:= ::::freebsd14:aarch64
port_checkout_unclean: no
port_git_hash=C2=A0 : f45cd5bd9d4b
ports_top_checkout_unclean: yes
ports_top_git_hash: 880f72e54deb


=3D=3D=3D
Mark Millard
marklmi at yahoo.co

This looks a bit like a= n issue I was hitting on a 4 CPU, 4 thread Alder Lake processor and 500GB S= SD running 13.2-RELEASE.

I saw several v= ery strange corruptions, at least one "rsync warning: some files vanis= hed before they could be transferred". In one (the last) case, the sys= tem crashed. The 'disc' was corrupted badly enough that fsck failed= and I could not boot up the system. The disk was UFS2 GPT format and EFI b= oot. The interface is SATA, not nVME. In this case, I was installing on a n= ew system and copying the majority of the file system from my old system. <= br>

The 'fix' is strange and pro= bably not one many other can use. I installed a spinning rust drive of 500G= B and installed FreeBSD and used rsync again and it worked. I can't say= whether it was a fluke that it worked, but it really smells like some sort= of race condition. Could be rsync , VFS, or device driver. Since then I ha= ve seen one crash while backing up the system disk using rsync. No corrupti= on and doing another rsync after reboot worked fine, but it was a much smal= ler run as the first attempt was nearly complete when the system crashed. M= aybe unrelated. I do have the core file from the crash. Stil, something wei= rd has been going on. Same issue on two identical systems, so not likely ha= rdware.
--
Kevin Oberman, Part ti= me kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
P= GP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
<= /div>
--000000000000b7b2ec0600a48f90--