Performance of madvise / msync

[Matthew Dillon]

:   65074 python   0.000006 CALL  madvise(0x287c5000,0x70,_MADV_WILLNEED)
:   65074 python   0.027455 RET   madvise 0
:   65074 python   0.000058 CALL  madvise(0x287c5000,0x1c20,_MADV_WILLNEED)
:   65074 python   0.016904 RET   madvise 0
:   65074 python   0.000179 CALL  madvise(0x287c6000,0x1950,_MADV_WILLNEED)
:   65074 python   0.008629 RET   madvise 0
:   65074 python   0.000040 CALL  madvise(0x287c8000,0x8,_MADV_WILLNEED)
:   65074 python   0.004173 RET   madvise 0
:...
:   65074 python   0.006084 CALL  msync(0x287c5000,0x453b7618,MS_ASYNC)
:   65074 python   0.106284 RET   msync 0
:...
:As you can see, it's quite a bit faster.
:
:I know that msync is necessary under Linux but obsolete under FreeBSD, but
:it's still funny that it takes a tenth of a second to return even with
:MS_ASYNC specified.
:
:Also, why is it that the madvise() calls take so much longer when the
:program does a couple of its own madvise() calls?  Was madvise() never
:intended to be run so frequently and is therefore a little slower than
:it could be?
:
:Here's the diff between the code for the first kdump above and the
:second one.

     Those times are way way too large, even with other running threads
     in the system.  madvise() should not take that long unless it is
     being forced to wait on a busied page, and neither should msync().
     madvise() doesn't even do any I/O (or shouldn't anyhow).

     Try removing just the msync() but keep the madvise() calls and see
     if the madvise() calls continue to take horrendous amounts of time.
     Then try the vise-versa.

     It kinda feels like a prior msync() is initiating physical I/O on
     pages and a later mmap/madvise or page fault is being forced to
     wait on the prior pages for the I/O to finish.

     The size_t argument to msync() (0x453b7618) is highly questionable.
     It could be ktrace reporting the wrong value, but maybe not.
     On any sort of random writing test, particularly if multiple threads
     are involved, specifying a size that large could result in very large
     latencies.

					-Matt