Re: Error detection for microSD-based swap, buildworld failures on pi3

On 2022-Feb-1, at 18:52, MJ <mafsys1234@gmail.com> wrote:

> On 2/02/2022 12:25 pm, Mark Millard wrote:
>> On 2022-Feb-1, at 16:47, MJ <mafsys1234@gmail.com> wrote:
>>> On 2/02/2022 3:18 am, bob prohaska wrote:
>>>> [new subject, different emphasis, old problem]
>>>> On Mon, Jan 31, 2022 at 03:06:01PM -0800, Mark Millard wrote:
>>>>> 
>>>>> One thing that could fit the behavior is if small part(s)
>>>>> of the system c++ compiler (or libraires it uses) were
>>>>> corrupted on that specific media. In that case, nothing
>>>>> elsewhere would replicate the failures but a lot might
>>>>> work without using the corrupted part(s), making the
>>>>> failures not random.
>>>> [spaced for emphasis]
>>>>> Checking on that is part of why
>>>>> I'd hoped to get a lldb report for a .sh/.cpp pair
>>>>> leading to failure on your RPi3* in question.
>>>>> 
>>>> If/when the stable/13 Pi3 finishes its -j1 single-user
>>>> build/install cycle I'll make a point of trying the
>>>> .sh/.cpp test under lldb.
>>>> For most of their operational history both troublesome Pi3
>>>> systems have had some of their swap on microSD. If there
>>>> is no error detection at all for microSD-based storage
>>> 
>>> Is this true? I would have thought it used some form of error detection in the firmware or in
>>> the controller.
>> The type of error and stage at which the error occurs matters.
>> The firmware can not cover all issues that lead to corrupted
>> content on media.
> 
> I did not state it covers all corruption. However, I would be totally surprised if the controller in
> ALL SD cards does not do error checking, whether ECC or even BCH. That remains my point.

I've a lot more context on Bob's problem and was mostly
giving more context than he supplied. (My name is on the
most nested text that he sent. I've been working with him
for a while on this.)

The corrupted-data hypothesis is a potential explanation
for the compiler "139" (SEGV) failures on the RPi3*
configuration in question for a few specific files being
compiled during buildworld. But we have no specific
evidence of a corruption at any specific place at this
point.

We have no specific evidence of a media-error type of
corruption either. The symptoms make it unlikely that
the swap partition pages are involved, block(s) in the
compiler's file(s) would be more likely. But, again, no
specific evidence identifying any specific block on the
media.

>>>> then undetected corruption of data from swap is a real
>>>> possibility. I expected that storage errors would be
>>>> reported but maybe not, especially outside file systems.
>>> 
>>> If indeed your suppositions are correct, would a file for swap be more prudent as it has to
>>> go through the file system (UFS/VFS) to read/write to swap?
>> No. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206048 and
>> its comments #7 and #8.
> 
> This seems to address potential memory over-use because of a swapfile, not the safety of it over a
> swap partition.

The problem is system deadlocks and such. A deadlock and having
to cut power and reapply power is a file system safety issue.

> I still contend the UFS file system has better protection against corruption than
> a raw partition labelled swap.

Not when the likely result is system deadlocks. I speak
from experience with trying the file system (v-node)
based page files.

> If Bob's requirement is a "safer" swap, then a file would be the answer.

My experience with the deadlocks and their consequences
indicate otherwise. That is why I added comments #7 and
#8 to that bugzilla submittal.

> Whether there are other issues to contend with are likely out of context in this particular discussion.

The deadlock consequences I suffered would most definitely
matter: starting over from scratch from something that
could not readily be recovered after the deadlock. The
deadlocks violate what guarantees file system update
coherency. (UFS for me back then and for Bob now.)
buildworld and the like tend to have a lot of pending
I/O around and a deadlock during this has not gone well
in my experience.

>>>> Mechanical disks have some internal error detection and
>>>> report explictly when data can't be retrieved. As I think
>>>> back on it at least one flash device (a USB thumb drive)
>>>> failed silently, no reported errors but also no-write.
>>>> That was on a filesystem, so the OS noticed and so did I.
>>> 
>>> But this could "simply" be because one of the NAND blocks has failed, not that it could not
>>> detect an error. Is there a lack of error detection in the driver handling USB thumb drives and reported back to the kernel? I do not know.
>> Bob's context is reproducible at the same places in
> 
> No, he was talking about a "failed silently" event and this is what I was replying to.

I've been working with Bob for some time on the issue
and have a lot of context on how to interpret things
that are not clear from what he extracted and sent out
of our off-list exchanges.

At this point it is hard for me to write notes without
using that context. This likely makes for an odd read
and unintended implications relative to only having
what Bob sent to the list.

We have no specific evidence of a media error at any
specific block/page. We have had no evidence of random
variations in the behavior beyond the expected sorts
of things from the likes of a -j4 buildworld : which
file was processed first being the one to get the
report of the compiler problem.

> I am not up-to-date with the previous discussion on the failure of llvm/clang.

I am. Some of it has been off-list. The corrupted data
hypothesis is a potential explanation for the compiler
"139" (SEGV) failures on the RPi3* configuration in
question.

>> Such is unlikely for hitting the same problem page(s)
>> in the swap space each way things are run.
> 
> I couldn't agree more. The chances would seem remote, unless that partition is on a part of the SD card/USB drive that is failing and the USB driver is not detecting these as reported by the controller.


===
Mark Millard
marklmi at yahoo.com