From nobody Sat Nov 16 12:31:25 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XrCtS2CWTz5dGdD for ; Sat, 16 Nov 2024 12:31:28 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XrCtS1bBFz55Mc; Sat, 16 Nov 2024 12:31:28 +0000 (UTC) (envelope-from avg@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731760288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=+eCoG+zzC1Ous1ah1Xjcs728Nhn81uCz5wHWIrEAlv0=; b=oGsVUslpBGyZhjoCn00kJaGKE3w8VPoImRHeIMGzeSsHubSTNDlRFOdgxBAPj+9bGy5GS+ cbS876r/wn8d8iopzG4YBRqSW9hY9HTAtZFga8LF/FMqMhsiGCT02pXT5QfiJdRpFBDqR1 5fmNFsRq1CxnN3N5VNT9DddWjF0OMpM3BHaTjcm72YaZIFWKDIDsgBQaUXte36gaAVunOc XRW8ZQCR45Wntd/7GmcQJZpWt2I+lZEyZU2Msn1v2ED3U/4acIVfrGi8yNylNTFYca6req DSfjIeFL5FgMvNZpb+rauoTS9+LZXYOnGIoDXildeZBjGdPsfuglc6sZfMlzOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731760288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=+eCoG+zzC1Ous1ah1Xjcs728Nhn81uCz5wHWIrEAlv0=; b=HHHZ8nueaZAzeY8B4OFp/ok4uUFNLVLKbxIva4ILoMYokF+tWbgMFB1qV3CLOEMI3OnWeh Orsc/erOQkCaGkvBVcG6rFbayxmuJQFTs4j33ZZj5EXBLOWVzp/QCId+aADNY7j6QhCFG0 2QEdpSpQN1iWUNDot8u84192W//4dzSCfh69jGS81LrgvKG5w3eEx9rlX85MBigawcKtdN 7NLOk+NKwCqFKMkibQ1lXyUvlvNT3RhRXhRZifgSygR6jpCFvW7km+EgRIcb/0x49LzY3o A6UyS2y4QN0f/F5ne4+XclAfnPTsbIa9cHt1eTA+tJnxgNaosGHpjFT2V53m3g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731760288; a=rsa-sha256; cv=none; b=MurrynSjVecYzLnKLWKZisdAKxkIK3fDfsTubTsOKIfjSGR9Bpg7X8U5Nn9EhDyKEA4b00 BYpXS3EM2SG+hhAPZUeFhv6VIQOlnGfVmSqvVIEZQzvf4vO1MFMOK2eHAm/WIVgTa6AV+U PcmyOPgatHQCBkMg2fzAFCYQ6bnTsz7YrjIAobQAnq/DiykmBlq7j2xY/mkYIIIDQ6w9lw TdedEykzJ0Zm+jw2wibYcywcUzjWve1IQHdHBGFGmK+YMuMG0CqA92GWgZYp6fzDrlm2q2 DqLmY6dLpTnvuHLg9R2rYWk/Za1maQwF+irquITrZxWlzJLZryY8MGWnynpR1A== Received: from [192.168.0.88] (unknown [93.188.39.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XrCtR5rwJzhMn; Sat, 16 Nov 2024 12:31:27 +0000 (UTC) (envelope-from avg@freebsd.org) Message-ID: <140eb994-ae19-41b4-8f0e-fc4290603ce0@freebsd.org> Date: Sat, 16 Nov 2024 14:31:25 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Andriy Gapon Subject: Re: tmpfs loses (sub-page chunks of) data? To: Konstantin Belousov Cc: freebsd-fs References: Content-Language: en-US Autocrypt: addr=avg@freebsd.org; keydata= xsDNBGcKrHEBDADRvwQOK0b/yo4ys5cs6bOQMhEh4xtfbaZ/CU00cpPgUip3sOZCdrtMWlRC g25z97prxE9pKueZi+HXDhIPpa9xl14ghqF4oYScuJ1i18HyiOH2y5Q3Vv/TtFiSzicd3EAu QgS3jVidpgDSPDdj2Yz3UxYpZ+PuFl6nOnvCvqOFcjUlzKCyPaiN2b86l1Nscmhnc+zQ/faB erUOEFEDQbWMA5YfXi8HrbeR16hfRfGt7E0aMDlIj9FIPIq71UWMN9CimPgs4+rbNr1MAlLa z4GxSDhVYZEY5rqtCzr+PLXboRQWnaUwXl0/biw9enf17NHdYv1SNAFTX2eC4dZ3qBVI74dS PgNprm+PMfz+6Hhs/dAv+Nan5nVhg3EFIjYTiy0MnjMSq8uI0v0ykpAGAcJJ5xl6d23aLxgN 6f0z6pJRCO0hGPgU7UzvFD0MxJxmbzqdT1R51KDan1oD41b+tjl2LMBuCDCoB0U44Pu0zLdp xMfFTxCXtwIYKIUxwd28jwMAEQEAAc0eQW5kcml5IEdhcG9uIDxhdmdARnJlZUJTRC5vcmc+ wsENBBMBCAA3FiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHEFCQeEzgACGwMECwkIBwUV CAkKCwUWAgMBAAAKCRDMM63k0uPru5tSDACFK15LLbq89RSQ6QMnjiIm1t/wYJyumb519MHu Dhzxx1lbr8oghf0RHtF6kYRLQPaW2VdToi74pRobd3CN4bhZKDLSL6WfTn17RfavDjL6Njwp KBo30CkOeYKWq1mDmo0xEoQj8cc7ybEZnus+YScZOpj8Ti4EFwhRt6SHer7YDb161IHKL8m4 MsCxpFSGEjbKj8Iul3Ri/fTOO8w14ivcuEEQIvJt4/+4YV5Az8G23wKzL/3aJ7SOT3oYGmR9 atBTmVO3DlODjM+rZLegd8SfLSPTcBTHspWE5duemIzZbEX3BP77r3Qx4Fo5Tkit3bG1XVar yPQato+sFGFEGifdE9USBQoAoOaaeZevwAWjDU0TIuCT0CUe0sKtQuNP4LRq0n9EEHOXBu9a CfdMhFUSkAZnuE7miSVwgPvoVNJ1stA37EXLN/sVsWik7wslTQ5vF81VpdGFiwoQPOe2XEKh ogcwGSnXbwv1gD4x+Gz/7Y+kFyr1NY+4/nSaeXVcS2fOwM0EZwqscgEMAMQTe6ypAmQe/TFO HqKD2hfFKdksTptKi6uEh8xIwct8G/0FBldDWXo9eu8CGr/ZrDg0/bAwJxbaLRQCMH19Gq2Y hLvZ1QK5GQJVzZKcqfxbF2LiDUTs6WkdOBIhGpdDy7p1xFrvqCGCtNFYHuGYm067EozibBSF BWAPstKu2FQuVHZNMOfs7p3OIz3Yfqu9woXDeg3/8G2qVQJINe+8EwXKlhgh4CyDbq7nAZoA kIu1SE9z9u3WI5mcNy/0dFmVUsFxBqRC3ewbvzie8tKyZ9yFOlaZPT0Y4nRBXQTI3mLZ8zQ8 mtrWK5OOmrJ02kdeO9RBXe+OMaUUWMf92ZIoBFb4HP6N+B+4N1y1OwULousfl7JRoYxA4MRL ls7E2sSoJvrEBTJB3Pc34xu8rsJ1A5V3NgN6djX8yEZYpTRkcmrBeWy/ofDqZPVqneAx0LRm eldDS9msXDW4KXODyPZ+9unvmHAcoH0xaBYaSH44CDZDQDg4LNcmbOvuu1TEXBJhjQARAQAB wsD8BBgBCAAmFiEEmXvSmjiQFHPVOpLnzDOt5NLj67sFAmcKrHMFCQeEzgACGwwACgkQzDOt 5NLj67sUCAv5AXqgWnYN9EblapMbZjkiqL8pZQ0GNqh+Pg9FwbyULxjtRTO6rD4D0IxizByb ef+neeUNyYlagt5nfKMysEr0SU/gHKCi8vyTF/63ukMrGUNGmJJxrndl5ZYKC6j6eX7twrZF L1Uvlmn6FnQ22red5kHO93fDjG4zaDIZvHfwj7kzjZ4tpC7Byinf88s14mdZeScc0PnU2hj4 UGYju/wg2FF4YxaZYhcmdTiRYY0Wx85XSMZv19pnn78sadEuRvfRd4JTmw++j1xGXeqQGWzz /CTG5/Ex9GAkQ02hZbmi236byDXoet4G8TEyOph9QFVkV9bNd0jQZaFZPGEj4PSPUYGAF7s5 xJaNGgctC3aZ7WjEv1FBoo44XCU4xcjJ1wZQUrHxRhx6TW0Jtcl0U9qfKFW30TSPo6RyiXuj X4ltWKAtjoXB8nUmEJckaz7IRu2b4pXDeazZuz5JBygUs10yJjDxh2vFQZo0KaBAPx9MZlPn gpPTjT15L8xGftEjQXF6 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 16/11/2024 00:24, Konstantin Belousov wrote: > On Fri, Nov 15, 2024 at 02:43:22PM +0200, Andriy Gapon wrote: >> >> We have a number of servers based on FreeSBD 13.3 that initially write some >> data to files on tmpfs and then the files are dispatched elsewhere. The >> writes are done by appending variable sized records to a file. There are no >> seeks or overwrites. >> >> I observe that occasionally (very rarely indeed given the amount of data >> produced) we get a corrupted file. >> >> In all cases so far the corruption follows the same pattern: data range from >> the end of a record until the next page-aligned boundary is zeroed out. >> That is, good data always continues from an offset which is multiple of 4096 >> and the zeroed area never crosses such offsets. >> >> Because of the page boundary, I have a suspicion that either tmpfs or, >> perhaps, the broader VM subsystem might have a race where writing to a page >> does not mark it dirty. Maybe this is related to paging out of a tmpfs page >> to the swap. >> >> The problem is that I have never been able to observe this happening, the >> corruption gets detected after the fact, hours after it occurs. >> >> If anyone could suggest any areas / changes / techniques to explore the >> problem, I would be much obliged. > > Do you have swap enabled on the problematic machines? Yes. > Are the files mapped, do you write or read through map? I think that mmap is not involved at all. Files are written from kernel using kern_writev(). After they are complete, they are read from userland using whatever libssl uses to read input files (when encrypting). Looks like that's fread(3) / read(2). -- Andriy Gapon