2 processes reproducible read same file with different speed

[Michael Powell]

cronfy wrote:

> Hello.
> 
> I've noticed a very weird behavior of 2 Apache processes that shold read
> the same file to process a request (they configured to read it on every
> request). One spends about 6ms to read the file, and second spends about
> 114ms (I used ktrace to find this out). Every time, on every request,
> the problem is reproducible. Apaches are the same, the only difference
> between them that they are working from different users to serve
> different sites. Same binary, same config.
> 
> First Apache used to work in the same way some time ago - it spent
> ~120ms to read the file. But once it changed and now it is working fast.
> 
> Restarts of Apache do not look to affect on anything.
> 
> The file that Apache should read is 315k long. Apache reads it by small
> blocks of 4096 bytes each. May be FreeBSD has some memory about how
> process is working with files and after some time enables some
> optimization or caching?.... I just do not have any clue... :(
> 
> Can anyone explain this please?
> 

Caching is coming into play if the first read takes longer than subsequent 
reads. It may not be uniform as older objects move in and out of cache being 
replaced by newer. The OS will have some space for buffer caching and when 
Apache makes a 'hit' on this cache retrieval will happen faster. This buffer 
space is dynamic in nature, and will shrink as Apache fills up memory. As 
Apache needs more memory the OS will attempt to provide as much as it can 
within reason by shrinking the buffer allocation space.

If this is a static file which needs to be read by all pages served, some 
other form of caching should be investigated. In the meantime you may want 
to try the following in httpd.conf if not already doing so:

EnableMMAP on
EnableSendfile on

You may also want to see if you can set aside a little RAM and cache it 
there, ala:

http://httpd.apache.org/docs/2.2/mod/mod_mem_cache.html

Another approach is a reverse proxy such as Varnish in front of the web 
server. But the first most immediate thing you can try and measure is to 
turn on sendile() if you're not using it.

-Mike