Re: 15-aarch64-RPI-snap

Reply: Mark Millard : "Re: 15-aarch64-RPI-snap"
In reply to: Glen Barber : "Re: 15-aarch64-RPI-snap"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 28 Oct 2023 05:24:29 UTC
On Oct 27, 2023, at 21:34, Glen Barber <gjb@FreeBSD.org> wrote:

> Relevant output from the failure follows.
> 
> Glen
> Sent from my phone.
> Please excuse my brevity and/or typos.
> 
>> On Oct 26, 2023, at 2:12 AM, Colin Percival <cperciva@tarsnap.com> wrote:
>> 
>> Err... that's an interesting failure...
>> 
>> Disk corruption?
>> 
>> Colin Percival
>> 
>>> On 10/25/23 23:05, root@FreeBSD.org wrote:
>>>                                                                                                        ^
>>> ./offset.inc:16:15: error: null character ignored [-Werror,-Wnull-character]
>>> <U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+00
>>> 00><U+0000>#undef _SA
>>>                                                                                                                ^
>>> ./offset.inc:16:16: error: null character ignored [-Werror,-Wnull-character]
>>> <U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+00
>>> 00><U+0000>#undef _SA
>>>                                                                                                                        ^
>>> ./offset.inc:16:17: error: null character ignored [-Werror,-Wnull-character]
>>> <U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+00
>>> 00><U+0000>#undef _SA
>>>                                                                                                                                ^
>>> ./offset.inc:16:18: error: null character ignored [-Werror,-Wnull-character]
>>> <U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+00
>>> 00><U+0000>#undef _SA
>>>                                                                                                                                        ^
>>> ./offset.inc:16:19: error: null character ignored [-Werror,-Wnull-character]
>>> <U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+00
>>> 00><U+0000>#undef _SA
>>>                                                                                                                                                ^

Are the above from a ZFS file system? UFS? Something else?

Back in 2021-Nov (15..21) I had problems where ZFS was leading
to blocks of such on aarch64, not specifically RPi*'s, various
files but not the same ones from test to test. When I updated
past some zfs updates on the 23rd the problem stopped.

I also have notes from 2022-Mar (19..22) about replicating
another example problem someone was having with files ending
up with such blocks of bytes but the testing was on the
ThreadRipper 1950X. (The replication showed that ccache did
not need to be involved since I've never used it.) Again
ZFS was part of the environment that got the replication.
Mark Johnson fixed sys/contrib/openzfs/module/zfs/dnode.c
during this and my ability to replicate the issue then
stopped when I tested the patch.

Which ever file system it is that holds the bad bytes, some
attempted testing for repeatability of the problem could
be of interest, some of that being on aarch64 but not on
RPi*'s, some of it not on aarch64 at all. But it might take
information about the context to know better what/how to
test. That could include information about both the host and
the jail OS versions if such is involved.


>>> fatal error: too many errors emitted, stopping now [-ferror-limit=]
>>> 20 errors generated.
>>> *** Error code 1
>>> Stop.
>>> make[5]: stopped in /usr/src/sys/modules/cxgbe/if_cxgbe
>>> *** Error code 1
>>> Stop.
>>> make[4]: stopped in /usr/src/sys/modules/cxgbe
>>> *** Error code 1
>>> Stop.
>>> make[3]: stopped in /usr/src/sys/modules
>>> *** Error code 1
>>> Stop.
>>> make[2]: stopped in /usr/obj/usr/src/arm64.aarch64/sys/GENERIC
>>>     1115.66 real       961.57 user        60.54 sys
>>> *** Error code 1
>>> Stop.
>>> make[1]: stopped in /usr/src
>>> *** Error code 1
>>> Stop.
>>> make: stopped in /usr/src
>>> rm -rf /R/ftp-stage
>>> sending incremental file list
>>> rsync: [sender] change_dir "/releng/15-aarch64-RPI-snap/R/ftp-stage/snapshots" failed: No such file or directory (2)
>>> sent 19 bytes  received 12 bytes  62.00 bytes/sec
>>> total size is 0  speedup is 0.00
>>> rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]
>> 
>> -- 
>> Colin Percival
>> FreeBSD Deputy Release Engineer & EC2 platform maintainer
>> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
> 


===
Mark Millard
marklmi at yahoo.com