madvise() vs posix_fadvise()

[John Baldwin]

On Friday, April 04, 2014 1:52:09 pm Dmitry Sivachenko wrote:
> 
> On 03 апр. 2014 г., at 19:02, John Baldwin <jhb at FreeBSD.org> wrote:
> 
> >> 
> >> Right now I am facing the following problem (stable/10):
> >> There is a (home-grown) webserver which mmap's a large amount of data files (total size is a bit below of RAM, say ~90GB of files with 128GB of RAM).
> >> Server writes access.log (several gigabytes per day).
> >> 
> >> Some of mmaped data files are used frequently, some are used rarely. On startup, server walks through all of these data files so it's content is read 
> > from disk.
> >> 
> >> After some time of running, I see that rarely used data files are purged from RAM (access to them leads to long-running disk reads) in favour of disk 
> > cache
> >> (at 0:00, when I rotate and gzip log file I see Inactive memory goes down to the value of log file size).
> >> 
> >> Is there any way to tell VM system not to push mmap'ed regions out of RAM in favour of disk caches?
> > 
> > Use POSIX_FADV_NOREUSE with fadvise() for the log files.  They are a perfect
> > use case for this flag.  This will tell the VM system to throw the log data
> > (move it to cache) after it writes the file.
> 
> 
> 
> Another question is why madvise(MADV_WILLNEED) is not enough to prefer
> keeping mmap'ed data in memory instead of dedicating all memory to cache log files?
> Even if that mmap'ed memory is rarely used.

MADV_WILLNEED is an instant action, not a policy statement.  It means
"please pre-fetch this data right now as I'm going to use it in the
next few seconds".  It does not mean "this data is more frequently
used, so try to keep it around for the next few hours".

> While POSIX_FADV_NOREUSE might be a solution for some cases (I am already
> testing it), it needs to be implemented in many programs (all that read/write
> files on disk), while madvise(MADV_WILLNEED) sounds like a proper solution
> to increase priority for mmaped region regardless of what other programs use
> disk but it does not seem to work as expected.

MADV_WILLNEED is not going to give you what you want.  OTOH, if you haven't
tried FreeBSD 10 yet, I would suggest trying that.  There have been changes
to pagedaemon that might make it do a better job of kicking out the pages
of the log files automatically.

-- 
John Baldwin