Re: tmpfs loses (sub-page chunks of) data?

On 16/11/2024 00:24, Konstantin Belousov wrote:
> On Fri, Nov 15, 2024 at 02:43:22PM +0200, Andriy Gapon wrote:
>>
>> We have a number of servers based on FreeSBD 13.3 that initially write some
>> data to files on tmpfs and then the files are dispatched elsewhere.  The
>> writes are done by appending variable sized records to a file.  There are no
>> seeks or overwrites.
>>
>> I observe that occasionally (very rarely indeed given the amount of data
>> produced) we get a corrupted file.
>>
>> In all cases so far the corruption follows the same pattern: data range from
>> the end of a record until the next page-aligned boundary is zeroed out.
>> That is, good data always continues from an offset which is multiple of 4096
>> and the zeroed area never crosses such offsets.
>>
>> Because of the page boundary, I have a suspicion that either tmpfs or,
>> perhaps, the broader VM subsystem might have a race where writing to a page
>> does not mark it dirty.  Maybe this is related to paging out of a tmpfs page
>> to the swap.
>>
>> The problem is that I have never been able to observe this happening, the
>> corruption gets detected after the fact, hours after it occurs.
>>
>> If anyone could suggest any areas / changes / techniques to explore the
>> problem, I would be much obliged.
> 
> Do you have swap enabled on the problematic machines?

Yes.

> Are the files mapped, do you write or read through map?

I think  that mmap is not involved at all.
Files are written from kernel using kern_writev().
After they are complete, they are read from userland using whatever libssl uses 
to read input files (when encrypting).  Looks like that's fread(3) / read(2).

-- 
Andriy Gapon