[PATCH] fadvise(2) system call

[Alfred Perlstein]

* John Baldwin <jhb at freebsd.org> [111108 05:29] wrote:
> On Friday, November 04, 2011 12:03:20 pm Alfred Perlstein wrote:
> > * John Baldwin <jhb at FreeBSD.org> [111103 19:38] wrote:
> > > On 11/2/11 1:20 PM, Alfred Perlstein wrote:
> > > >* John Baldwin<jhb at freebsd.org>  [111102 11:05] wrote:
> > > >>On Tuesday, November 01, 2011 7:56:48 pm Bruce Cran wrote:
> > > >>>On 01/11/2011 17:08, John Baldwin wrote:
> > > >>>>I had found it via the web: http://linux.die.net/man/2/fadvise
> > > >>>>However, after further searching it appears to be stale (if you follow
> > > >>>>it's cross-reference to madvise(2), that page only has links to
> > > >>>>posix_fadvise() and not fadvise()).
> > > >>>
> > > >>>There's
> > > >>>http://www.speedware.com/HPe3000_resources/MPE_to_HP-UX_cross-
> > > >>reference/system_administration_cross-reference/cmd.html?cmdid=MS_1800
> > > >>>for HP-UX ("*fadvise()* was derived by HP from the IEEE POSIX
> > > >>>1003.1-2001 Standard"), though it also has posix_fadvise.
> > > >>
> > > >>Hmm, that one actually has an extra argument.  I'll just go with
> > > >>posix_fadvise() for now.  Interesting that HP lets you OR together
> > > >>two policies (so you can say both "I will access this file sequentially
> > > >>and with noreuse").
> > > >
> > > >Makes sense for gzip/tar.
> > > 
> > > Ehh, quite possibly not for something that generic.  I think you only
> > > want to do this if you have very specific knowledge about your access
> > > pattern, and I do not think they are generically applicable.
> > 
> > You often spend time untarring the same tarball over and over 
> > in your workflow John?
> 
> No, but many people will do a 'tar tvf' of a file after they generate it.  

That's not the use case I was speaking of.  I was speaking of actually
extracting the file as I said.

One other optimization would be to set the bit when tar or other such apps
notice that the file exceeds the memory of the system, effectively noting
that it would be blowing out the entire cache.

I don't think there's a need to respond to the rest of this so I will not.

> Would be rather silly to force that to go to disk every time, especially given 
> tar's format where it does not have a header like zip so you have to read the 
> whole darn thing for a 'tar tvf'.
> 
> I think it would be fine to add flags to applications like 'tar' to allow 
> users to alter their behavior in specific use cases when it makes sense.  
> However, I think there are more workloads for 'tar' than the ones you are 
> thinking of and we should be hesitant to change applications to use non-
> default settings.  I do think FADV_SEQUENTIAL would be easy to add hints for, 
> it is more FADV_NOREUSE and FADV_DONTNEED that I think warrant more caution as 
> you can end up greatly hampering performance if you are not careful.  I think 
> for the general case we should let pagedaemon manage this instead.  This is 
> similar to the discussions about NUMA and the various notes from people who 
> have worked with it in the past that having the kernel try to employ 
> heuristisc often ends up breaking many things vs letting non-default behavior 
> being driven by hints from users.

-- 
- Alfred Perlstein
.- VMOA #5191, 03 vmax, 92 gs500, 85 ch250, 07 zx10
.- FreeBSD committer