posix_fadvise noreuse disables file caching

[John Baldwin]

On Thursday, January 19, 2012 11:39:42 am Tijl Coosemans wrote:
> Hi,
> 
> I recently noticed that multimedia/vlc generates a lot of disk IO when
> playing media files. For instance, when playing a 320kbps mp3 gstat
> reports about 1250kBps (=10000kbps). That's quite a lot of overhead.
> 
> It turns out that vlc sets POSIX_FADV_NOREUSE on the entire file and
> reads in chunks of 1028 bytes. FreeBSD implements NOREUSE as if
> O_DIRECT was specified during open(2), i.e. it disables all caching.
> That means every 1028 byte read turns into a 32KiB read (new default
> block size in 9.0) which explains the above numbers.
> 
> I've copied the relevant vlc code below (modules/access/file.c:Open()).
> It's interesting to see that on OSX it sets F_NOCACHE which disables
> caching too, but combined with F_RDAHEAD there's still read-ahead
> caching.
> 
> I don't think POSIX intended for NOREUSE to mean O_DIRECT. It should
> still cache data (and even do read-ahead if F_RDAHEAD is specified),
> and once data is fetched from the cache, it can be marked WONTNEED.

POSIX doesn't specify O_DIRECT, so it's not clear what it asks for.

> Is it possible to implement it this way, or if not to just ignore
> the NOREUSE hint for now?

I think it would be good to improve NOREUSE, though I had sort of
assumed that applications using NOREUSE would do their own buffering
and read full blocks.  We could perhaps reimplement NOREUSE by doing
the equivalent of POSIX_FADV_DONTNEED after each read to free buffers
and pages after the data is copied out to userland.  I also have an
XXX about whether or not NOREUSE should still allow read-ahead as it
isn't very clear what the right thing to do there is.  HP-UX (IIRC)
has an fadvise() that lets you specify multiple policies, so you
could specify both NOREUSE and SEQUENTIAL for a single region to
get read-ahead but still release memory once the data is read once.

-- 
John Baldwin