Why would it make such a difference to move mysqld to another machine?

[Mel]

On Sunday 06 July 2008 10:58:54 Matthew Seaman wrote:

> I suspect that you could have achieved a pretty good speed-up simply by
> adding another hard drive to your server and moving all of the database
> onto it, separate from the web root and any other areas which apache
> would be doing a lot of read/write operations on.

In fact, when you look at the two top(1) outputs from OP, you can see that 
apache is eating all the memory and hardly any memory is left for IO cache. 
On the second machine you can see that MySQL is caching 49MB I/O, guessing 
this is the (sum of) the most used indexes or in case of InnoDB the entire 
InnoDB file.
Since MySQL is clearly the bottleneck of the sites, I'd investigate why in the 
world apache2 needs >150M per process. If you're using the memcache module, 
it may actually speed things up considerably to *disable* it, since that 
would leave plenty IO cache for MySQL to work with.

On the other hand, if there's so much I/O to cache for the webserver, maybe 
it's a good idea to configure a lightweight image (or other static content) 
server. This apache would only need mod_mime, mod_header and mod_expires (to 
set the caching). Personally, I wouldn't enable the mem cache on there 
either, since 80% of the time you'll be sending 304's, instead of image data.
-- 
Mel

Problem with today's modular software: they start with the modules
    and never get to the software part.