Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS)

[Mikhail Teterin]

вівторок 28 березень 2006 05:27, Peter Jeremy написав:
> I'd suggest that each mmap be capable of storing several hundred msec of
> data as a minumum (maybe 10MB input and 5MB output, preferably more).
> Synchronisation can be done by writing tokens into pipes shared with the
> mmap's, optimised by sharing read/write pointers (so you only really need
> the tokens when the shared buffer is full/empty).

Thank you very much, Peter, for your suggestions. Unfortunately, I have no 
control whatsoever over the dump-ing part of the process. The dump is done by 
Sybase database servers -- old, clunky, and closed-source software, running 
on slow CPU (but good I/O) Sun hardware.

You are right, of course, that my application (mzip being only part of it) 
needs to keep the dumper and the compressor in sync. Without any cooperation 
from the former, however, I see no other way but to temporarily throttle the 
NFS-bandwidth via firewall, when the compressor falls behind (as can be 
detected by the increased proportion of sys-time, I guess).

Much as I apreciate the (past and future) help and suggestions, I'm not asking 
you, nor the mailing list to solve my particular problem here :-) I only gave 
the details of my need and application to illustrate a missed general 
optimization opportunity in FreeBSD -- reading large files via mmap need not 
be slower than via read. If anything, it should be (slightly) faster.

After many days Matt has finally stated (admitted? ;-):

	read() uses a different heuristic then mmap() to implement the
	read-ahead. There is also code in there which depresses the page
	priority of 'old' already-read pages in the sequential case.

There is no reason not to implement similar smarts in the mmap-handling code
to similarly depress the priority of the in-memory pages in the 
MADV_SEQUENTIAL case, thus freeing more RAM for aggressive read-ahead.

As I admitted before, actually implementing this far exceeds my own 
capabilities, so all I can do is pester, whoever cares, to do it instead :-)
C'mon, guys...

	-mi