Swap Issue

[Doug Hardie]

I have a most interesting situation that just manifested itself this morning in a way I could begin to diagnose.  The system runs 8.2-P3 and has no users, just one process that runs 24x7.  Its been running since 8.2 was first released.  Every now and then the process becomes non-responsive.  Today it managed to spit out an error that the system had run out of swap space before it hung.

This is quite interesting in that there are 5.5GB of free memory.  The only time that changes is at 3 am when the various periodic processes run.  However, I don’t suspect those as I have numerous other systems that have considerably less memory and higher usage that do not experience this issue.  I have increased the swap space to prevent this issue, but something must be going bonkers to cause the swap space to get used.  Usually there are only 15MB used.  Every time the system notifies me that it has the problem, by the time I can look at it, swap is back to normal.  I have never been able to catch the system with more than 15 MB used.  Is there any way after the fact to figure out what was using the swap?  I suspect not, but thought I’d ask just in case I have overlooked something.