Strange memory management with mmap()

[Andrey Zonov]

On 7/22/15 5:47 AM, Dmitry Sivachenko wrote:
> 
>> On 16 июля 2015 г., at 21:19, Dmitry Sivachenko <trtrmitya at gmail.com> wrote:
>>
>>>
>>> On 16 июля 2015 г., at 18:42, Dmitry Sivachenko <trtrmitya at gmail.com> wrote:
>>>
>>> Hello!
>>>
>>> I am using FreeBSD-10-stable and writing a program that uses large data file via mmap() in read only mode.
>>> To be specific, I have 256GB RAM machine and typical size of data file is ~160GB (more than 1/2 of RAM and less that the whole RAM).
>>> There is no other programs running during the test.
>>>
>>> Consider the following use case: I have two files on disk.  I mmap() the first one and prefetch data to RAM (touch every page of the file).
>>> After that I expect all data to be cached in RAM and subsequent access will be fast.
>>>
>>> Next I do munmap() on the first file, mmap() the second one and do the same test: prefetch data and expect it to be cached in RAM (and some of the pages belonging to the first file to be purged out, because size_of(file1)+size_of(file2) > size_of(RAM).
>>>
>>> Please find my test program attached.
>>>
>>> I run the program with 2 files provided via command line (both about 160GB).
>>> What I observe in real is:
>>> -- before I run the program all RAM is in FREE state as reported by top(1).
>>> -- after first prefetch() of the first file, all it's data goes to "Cache" state, RES column of the process remains the same (small)
>>> -- second prefetch() works fast as expected, memory goes from Cache to Active state, RES column of the process grows up to match file size (SIZE==RES now)
>>> -- now first prefetch() for second file starts: the remaining Free memory goes to Cache state, Active size still equals to first file size.
>>> -- second prefetch() for second file works as slow as first one, like if nothing was cached in memory during the first prefetch() run, RES column does not change.
>>>
>>>
>>> Here is the output:
>>> % /tmp/a.out file1.dat file2.dat
>>> file1.dat... First prefault time: 1235.747351 seconds
>>> Second prefault time: 74.893323 seconds
>>> Ok.
>>> file2.dat... First prefault time: 1316.405527 seconds
>>> Second prefault time: 1311.491842 seconds
>>> Ok.
>>>
>>
>>
> 
> 
> I tried the same test program on Linux machine with similar hardware.  It behaves like expected (second prefetch works very fast):
> 
> file1.dat... First prefault time: 2664.621088 seconds
> Second prefault time: 1.969283 seconds
> Ok.
> file2.dat... First prefault time: 2917.009003 seconds
> Second prefault time: 34.128762 seconds
> Ok.
> 

"Cache" works as a FIFO, so beginning of the second file is out of
"Cache" by the the end of first prefault().

Workaround is to do double prefault() for segments, e.g. 1Gb.


-- 
Andrey Zonov

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 521 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20160425/21eaffd8/attachment.sig>