Strange memory management with mmap()

[Dmitry Sivachenko]

Hello!

I am using FreeBSD-10-stable and writing a program that uses large data file via mmap() in read only mode.
To be specific, I have 256GB RAM machine and typical size of data file is ~160GB (more than 1/2 of RAM and less that the whole RAM).
There is no other programs running during the test.

Consider the following use case: I have two files on disk.  I mmap() the first one and prefetch data to RAM (touch every page of the file).
After that I expect all data to be cached in RAM and subsequent access will be fast.

Next I do munmap() on the first file, mmap() the second one and do the same test: prefetch data and expect it to be cached in RAM (and some of the pages belonging to the first file to be purged out, because size_of(file1)+size_of(file2) > size_of(RAM).

Please find my test program attached.

I run the program with 2 files provided via command line (both about 160GB).
What I observe in real is:
-- before I run the program all RAM is in FREE state as reported by top(1).
-- after first prefetch() of the first file, all it's data goes to "Cache" state, RES column of the process remains the same (small)
-- second prefetch() works fast as expected, memory goes from Cache to Active state, RES column of the process grows up to match file size (SIZE==RES now)
-- now first prefetch() for second file starts: the remaining Free memory goes to Cache state, Active size still equals to first file size.
-- second prefetch() for second file works as slow as first one, like if nothing was cached in memory during the first prefetch() run, RES column does not change.


Here is the output:
% /tmp/a.out file1.dat file2.dat
file1.dat... First prefault time: 1235.747351 seconds
Second prefault time: 74.893323 seconds
Ok.
file2.dat... First prefault time: 1316.405527 seconds
Second prefault time: 1311.491842 seconds
Ok.

I treat this like the bug somewhere in virtual memory management.  Am I right or my expectations about how that test program should work are false?

Thanks in advance.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap_test.c
Type: application/octet-stream
Size: 1759 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20150716/4ab6636a/attachment.obj>