Userland hangs on overloaded server

[Niklas Saers]

Hi all,

a server I run at work (a small ISP) has about 350 jails, one for each
customer. It runs a webserver of the customers choice and configuration,
contains all his data, and is a full FreeBSD default environment plus a
few apps like favorite editors and such.

Due to a few problems we had with 5.2, we decided to take the server up to
CURRENT until 5.3. The following problem has been from we installed the
CURRENT of March 30th to and including yesterday's source.

After having been running for a while, userland stops responding. All the
running daemons, webservers and inetd keep running. When I enter by SSH,
sshd will accept the connection and show motd, but then it will not give
me a shell, but simply hang. If this happens while I'm editing a file, I
can save my file fine and exit to the shell and get a prompt, but if I
press enter or type any command, it will not return until I've rebooted.

Does anyone have a suggestion of what may be going on? The logs tell me
nothing, so I haven't been running out of swap or anything similar.

For those interested, it's a dual Athlon MP (can't remember the exact
speed) with 2GB RAM.

Cheers

   Nik