Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS)
mi+mx at aldan.algebra.com
Tue Mar 28 18:27:57 UTC 2006
вівторок 28 березень 2006 05:27, Peter Jeremy написав:
> I'd suggest that each mmap be capable of storing several hundred msec of
> data as a minumum (maybe 10MB input and 5MB output, preferably more).
> Synchronisation can be done by writing tokens into pipes shared with the
> mmap's, optimised by sharing read/write pointers (so you only really need
> the tokens when the shared buffer is full/empty).
Thank you very much, Peter, for your suggestions. Unfortunately, I have no
control whatsoever over the dump-ing part of the process. The dump is done by
Sybase database servers -- old, clunky, and closed-source software, running
on slow CPU (but good I/O) Sun hardware.
You are right, of course, that my application (mzip being only part of it)
needs to keep the dumper and the compressor in sync. Without any cooperation
from the former, however, I see no other way but to temporarily throttle the
NFS-bandwidth via firewall, when the compressor falls behind (as can be
detected by the increased proportion of sys-time, I guess).
Much as I apreciate the (past and future) help and suggestions, I'm not asking
you, nor the mailing list to solve my particular problem here :-) I only gave
the details of my need and application to illustrate a missed general
optimization opportunity in FreeBSD -- reading large files via mmap need not
be slower than via read. If anything, it should be (slightly) faster.
After many days Matt has finally stated (admitted? ;-):
read() uses a different heuristic then mmap() to implement the
read-ahead. There is also code in there which depresses the page
priority of 'old' already-read pages in the sequential case.
There is no reason not to implement similar smarts in the mmap-handling code
to similarly depress the priority of the in-memory pages in the
MADV_SEQUENTIAL case, thus freeing more RAM for aggressive read-ahead.
As I admitted before, actually implementing this far exceeds my own
capabilities, so all I can do is pester, whoever cares, to do it instead :-)
More information about the freebsd-stable