vmstat memory: avm vs fre

[Marc G. Fournier]

I'm getting a really odd condition on one of my servers (and I suspect its 
happening on one of my other servers as well) ... after a period of time 
(<3 days), the server hangs solid ...

Running vmstat in an xterm, the one thing I'm noticing is that when it 
hangs, my avm == 12455M and fre == 22M ... when I start the system, it 
looks like: avm == 246M vs fre == 197M ...

I'm suspecting that the lock up is that fre hit 0 at some point, but I'm 
at a loss as to why, or where to look, for this ...

top in another xterm when it hangs shows it appears to have more then 
enough VM:

last pid: 87005;  load averages:  8.57,  7.29,  4.46    up 0+17:25:13  20:45:00
1140 processes:317 running, 774 sleeping, 10 zombie, 39 lock
CPU: 23.3% user,  0.0% nice, 11.1% system,  0.4% interrupt, 65.1% idle
Mem: 4610M Active, 440M Inact, 489M Wired, 13M Cache, 214M Buf, 9624K Free
Swap: 8192M Total, 1055M Used, 7137M Free, 12% Inuse, 564K In, 272K Out
kvm_open: cannot open /proc/90106/mem
   PID JID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU 
COMMAND
30625   0 root        1  96    0   588M   166M RUN    0  14:54  0.10% 
/usr/local/bin/qemu-system-x86_64 -m 512M -net nic,macadd
86866  20   1200      1  96    0 60888K  1140K RUN    0   0:00  0.15% 
postgres: autovacuum worker process    (postgres)
86844   1 root        1  96    0 15080K  1028K RUN    1   0:00  0.05% 
sshd: [accepted] (sshd)
45533  20 root        1  96    0 15044K   456K RUN    1   0:00  0.05% 
/usr/sbin/sshd
86895   0 root        1  96    0 15092K   428K RUN    0   0:00  0.05% 
/usr/sbin/sshd
15131  15 root        1  96    0 19692K   376K RUN    1   0:00  0.15% 
/usr/sbin/sshd
95911   4 www         1   4    0   106M     0K accept 0   0:01  0.00% 
/usr/local/sbin/httpd (<httpd>)


----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . scrappy at hub.org                              MSN . scrappy at hub.org
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664