MS_ASYNC with MS_INVALIDATE

[Peter Jeremy]

On Sun, Mar 28, 2004 at 03:02:37PM -0600, Dan Nelson wrote:
>In the last episode (Mar 27), Mark Terribile said:
>> A friend asked me to run some Linux source on FreeBSD.  It simulates
>> a data pool management system he is using, and it includes a call
>> msync(2) with both the MS_ASYNC and MS_INVALIDATE flags.  FreeBSD
>> does not allow this. (I tested it on my 4.8 system; I'll have access
>> to a 5.1 system on Monday.) I ran it without the MS_ASYNC, and got
>> very different results from the ones he reported on Linux.  I'd like
>> to be able to explain what's happening.

My interpretation of the man page suggests that MS_ASYNC should not
have any effect other than timing unless there are race conditions.

>> Would someone please shed some light on why this combination of flags
>> is disallowed, and what the intended semantics are?  I suspect that
>> either I or he has a subtle misunderstanding here.
>
>Does that even make sense?  How could you simultaneously request that
>all modified pages be queued for writing to disk AND that all modified
>pages be discarded?  If you discard them, then there's nothing for the
>kernel to write.

Based on the description in the man page, it's not at all clear why
MS_ASYNC and MS_INVALIDATE are mutually exclusive.  The description of
msync(2) is (in 4.9):
  The msync() system call writes any modified pages back to the filesystem
  and updates the file modification time.  If len is 0, all modified pages
  within the region containing addr will be flushed; if len is non-zero,
  only those pages containing addr and len-1 succeeding locations will be
  examined.  The flags argument may be specified as follows:

  MS_ASYNC        Return immediately
  MS_SYNC         Perform synchronous writes
  MS_INVALIDATE   Invalidate all cached data

The first sentence states that modified pages are written back to the
filesystem.  The description of MS_INVALIDATE does not state that the
modified pages are not written back - it would seem perfectly reasonable
to assume that MS_INVALIDATE wipes the cache _after_ the data has been
written (ie save the changes I've made but I won't be referring to this
data again so don't waste RAM remembering it).  It's not at all obvious
why this can't occur asynchronously - in fact, this would seem the most
logical combination of you've finished using the area in question.

MS_ASYNC and MS_SYNC would appear to be mutually exclusive, but this
isn't reported as an error (the implementation in 4.9 actually totally
ignores MS_SYNC).  Likewise, (MS_SYNC | MS_INVALIDATE) is allowed -
which only makes sense if the invalidation occurs after writing.

Peter