Strange memory management with mmap()

[RW]

On Thu, 16 Jul 2015 18:42:26 +0300
Dmitry Sivachenko wrote:


> I run the program with 2 files provided via command line (both about
> 160GB). What I observe in real is:
> -- before I run the program all RAM is in FREE state as reported by
> top(1). -- after first prefetch() of the first file, all it's data
> goes to "Cache" state, 

This is what surprises me most, I'd expect it to go to active. I thought
that the point of the cache queue is that its pages can be reassigned
because they are clean and not mapped anywhere. 


>RES column of the process remains the same
> (small) -- second prefetch() works fast as expected, memory goes from
> Cache to Active state, RES column of the process grows up to match
> file size (SIZE==RES now) -- now first prefetch() for second file
> starts: the remaining Free memory goes to Cache state, Active size
> still equals to first file size. -- second prefetch() for second file
> works as slow as first one, like if nothing was cached in memory
> during the first prefetch() run, RES column does not change.
>
> Here is the output:
> % /tmp/a.out file1.dat file2.dat
> file1.dat... First prefault time: 1235.747351 seconds
> Second prefault time: 74.893323 seconds
> Ok.
> file2.dat... First prefault time: 1316.405527 seconds
> Second prefault time: 1311.491842 seconds
> Ok.
> 
> I treat this like the bug somewhere in virtual memory management.

The timings do make sense. The management of the active queue isn't
LRU, it takes account of how many times a page gets accessed. Whatever
is going on with the cache queue doesn't seem to change that general
strategy.

The double prefail on the first file means that all its pages are in
memory and have recently been accessed twice. When the system runs out
of free memory the VM system retains the file1 pages at the expense of
the pages from file2 which have only been accessed once. At the end of
the first pass only the end of file2 is cached, which isn't any use on
the second pass because the prefault works upwards. 


This is an uncommon pattern of access where simple LRU expiry of the
cache will outperform anything more sophisticated. I don't think
it matters that Linux does better on this particular benchmark.