resource leak

[Stephen Clark]

Hello List,

I am running into a strange problem that points to a resource leak. The problem 
manifests itself after one of our remote systems has been up around 100 days.
The symptom is that it appears no new processes can be spawned. If I try to
ssh to the unit, I can see the 3-way tcp handshake and then no more traffic.
Examining log files, like cron, etc show that when this happens no more entries
are written into the cron log. The unit is acting as a firewall, router and vpn 
appliance these functions continue to work. We have a C application that is 
periodically started out of a shell script that reports various information 
about the system, it stops reporting, while vpns, ospf routing, and ipfilter 
firewalling continue to work and write into their logfiles.

My question is how do I monitor the various resources in the system that could
prevent the spawning of a new process?

This is on FreeBSD 6.1, ipsec-tools-6.6, quagga-0.99.3

Any ideas or directions would be  greatly appreciated.


Thanks,
Steve