madvise() vs posix_fadvise()

[Dmitry Sivachenko]

On 03 апр. 2014 г., at 19:02, John Baldwin <jhb at FreeBSD.org> wrote:

>> 
>> Right now I am facing the following problem (stable/10):
>> There is a (home-grown) webserver which mmap's a large amount of data files (total size is a bit below of RAM, say ~90GB of files with 128GB of RAM).
>> Server writes access.log (several gigabytes per day).
>> 
>> Some of mmaped data files are used frequently, some are used rarely. On startup, server walks through all of these data files so it's content is read 
> from disk.
>> 
>> After some time of running, I see that rarely used data files are purged from RAM (access to them leads to long-running disk reads) in favour of disk 
> cache
>> (at 0:00, when I rotate and gzip log file I see Inactive memory goes down to the value of log file size).
>> 
>> Is there any way to tell VM system not to push mmap'ed regions out of RAM in favour of disk caches?
> 
> Use POSIX_FADV_NOREUSE with fadvise() for the log files.  They are a perfect
> use case for this flag.  This will tell the VM system to throw the log data
> (move it to cache) after it writes the file.



Another question is why madvise(MADV_WILLNEED) is not enough to prefer keeping mmap'ed data in memory instead of dedicating all memory to cache log files?
Even if that mmap'ed memory is rarely used.

While POSIX_FADV_NOREUSE might be a solution for some cases (I am already testing it), it needs to be implemented in many programs (all that read/write files on disk), while
madvise(MADV_WILLNEED) sounds like a proper solution to increase priority for mmaped region regardless of what other programs use disk but it does not seem to work as expected.