posix_fadvise noreuse disables file caching

[John Baldwin]

On Tuesday, January 31, 2012 12:21:07 pm Ulrich Spörlein wrote:
> On Mon, 2012-01-30 at 09:36:45 -0500, John Baldwin wrote:
> > On Sunday, January 29, 2012 10:08:10 am Tijl Coosemans wrote:
> > > On Wednesday 25 January 2012 17:29:22 John Baldwin wrote:
> > > > On Friday, January 20, 2012 2:12:13 pm John Baldwin wrote:
> > > >> On Thursday, January 19, 2012 11:39:42 am Tijl Coosemans wrote:
> > > >>> I recently noticed that multimedia/vlc generates a lot of disk IO when
> > > >>> playing media files. For instance, when playing a 320kbps mp3 gstat
> > > >>> reports about 1250kBps (=10000kbps). That's quite a lot of overhead.
> > > >>> 
> > > >>> It turns out that vlc sets POSIX_FADV_NOREUSE on the entire file and
> > > >>> reads in chunks of 1028 bytes. FreeBSD implements NOREUSE as if
> > > >>> O_DIRECT was specified during open(2), i.e. it disables all caching.
> > > >>> That means every 1028 byte read turns into a 32KiB read (new default
> > > >>> block size in 9.0) which explains the above numbers.
> > > >>> 
> > > >>> I've copied the relevant vlc code below (modules/access/file.c:Open()).
> > > >>> It's interesting to see that on OSX it sets F_NOCACHE which disables
> > > >>> caching too, but combined with F_RDAHEAD there's still read-ahead
> > > >>> caching.
> > > >>> 
> > > >>> I don't think POSIX intended for NOREUSE to mean O_DIRECT. It should
> > > >>> still cache data (and even do read-ahead if F_RDAHEAD is specified),
> > > >>> and once data is fetched from the cache, it can be marked WONTNEED.
> > > >> 
> > > >> POSIX doesn't specify O_DIRECT, so it's not clear what it asks for.
> > > >> 
> > > >>> Is it possible to implement it this way, or if not to just ignore
> > > >>> the NOREUSE hint for now?
> > > >> 
> > > >> I think it would be good to improve NOREUSE, though I had sort of
> > > >> assumed that applications using NOREUSE would do their own buffering
> > > >> and read full blocks.  We could perhaps reimplement NOREUSE by doing
> > > >> the equivalent of POSIX_FADV_DONTNEED after each read to free buffers
> > > >> and pages after the data is copied out to userland.  I also have an
> > > >> XXX about whether or not NOREUSE should still allow read-ahead as it
> > > >> isn't very clear what the right thing to do there is.  HP-UX (IIRC)
> > > >> has an fadvise() that lets you specify multiple policies, so you
> > > >> could specify both NOREUSE and SEQUENTIAL for a single region to
> > > >> get read-ahead but still release memory once the data is read once.
> > > >
> > > > So I've came up with this untested patch.  It uses
> > > > VOP_ADVISE(FADV_DONTNEED) after read(2) calls to a NOREUSE region, and
> > > > leaves read-ahead caching enabled for NOREUSE.  FADV_DONTNEED doesn't
> > > > do any good really for writes (it only flushes clean buffers), so I've
> > > > left write(2) operations as using IO_DIRECT still.  Does this sound
> > > > reasonable?  I've not yet tested this at all:
> > > 
> > > The patch drastically improves vlc, but there's still a tiny overhead.
> > > Without NOREUSE the disk is read in chunks of 128KiB (F_RDAHEAD buffer
> > > size). With NOREUSE there's an extra transfer of 32KiB (block size).
> > 
> > This is probably because vlc is not reading on block boundaries, so the 
> > noreuse is throwing away partial blocks at the end of a read that then have to 
> > be re-read.  We could maybe fix this by making FADV_DONTNEED only throw
> > away completely-contained blocks rather than completely-contained pages.
> > However, this will probably result in NOREUSE not actually throwing away
> > anything at all if an app always reads sub-blocksize chunks.
> > 
> > We could maybe make the case of vlc work ok in this case though by allowing
> > an extension where you can do 'posix_fadvise(SEQUENTIAL | NOREUSE)', and
> > in this case we could make the VOP_ADVISE(DONTNEED) in read() use an offset
> > of 0 rather than the start of the read request.
> > 
> > However, posix_fadvise() really is going to work best if the userland 
> > application reads aligned FS blocks.
> 
> I find it questionable in general that an application can tell the
> system what to do wrt. caching. Perhaps I'm running 100s of VLC players
> all on the same file and actually *do* want reads to be cached?
> 
> What happens if I seek back in the file? It has to do a potentially
> high-latency read again. The system has a better overview of blocks that
> are frequently being requested than any individual application.
> 
> I fully understand the intention, and in 99.99% of the cases, this data
> *is* just being read once so there's no need to cache any reads for
> actually requested data. But as the example shows, requested data is not
> necessarily the data that lower layers have to fetch from the disk.
> 
> Perhaps taking to VLC people on why they think this is useful and where
> it actually, measurably helped them would be interesting.
> 
> Sorry if this is all perfectly obvious

There are certainly cases where the user can choose to run specific apps in
such a way where this makes sense, so the OS needs this functionality.  As
to whether or not specific apps should use these APIs or if they should make
use of these APIs configurable, that is a question for each app (e.g. vlc).
However, the OS should provide the tools.

-- 
John Baldwin