Dubious VM behavior scanning large files

[Thomas Hurst]

I'm currently verifying a 50GB filesystem dump with sha1 at about
40MB/s.  This is resulting in a small but annoying amount of swapping,
including my IRC client, text editors, even syslogd.  Larger processes
like my .5G-each mysqld and squid don't seem to be effected:

Mem: 2045M Active, 5025M Inact, 439M Wired, 251M Cache, 214M Buf, 11M Free
Swap: 10G Total, 12M Used, 10G Free, 232K Out

cat(1), dd(1), par2, all result in the same sort of thing; I have well
over over 4GB of memory used for nothing but disk cache, and the VM is
somehow deciding it's cheaper to swap out fairly active processes
(including syslogd) than it is to free up a few inactive pages.

Swap never goes above about 12MB, but pages are in and out fairly often.
Do I need to tune the VM to keep more free pages around?

This is a dual dual core AMD64 system with 8G of memory, and it's
running 6.2-STABLE as of July 4th, and verified just now with -STABLE
dated Sep 16.

I also again noticed dubious swap metering from top/swapinfo during
today's testing:

 1169 clamav        1   4    0 67620K     0K accept 2   0:04  0.00%
 <clamd>
 1301 freaky        1  96    0 24916K     0K select 3   0:00  0.00%
 <sshd>

Along with a few other processes, and yet:

  Swap: 10G Total, 8K Used, 10G Free

This has been mentioned before, and I don't think anyone was able to
explain it.  Swap later rose to about 900k, but only after several
instances of SpamAssassin were marked as swapped.

-- 
Thomas 'Freaky' Hurst
    http://hur.st/