ssh_exchange_identification: Connection closed by remote host

[John Almberg]

>>> If you can't log in -- even on the console -- then rebooting is really
>>> your only option.  Ctrl-Alt-Del should bring the system down cleanly if
>>> you haven't disabled that functionality.  Otherwise, just toggle the
>>> power.
>>>
>>> The symptoms you're seeing could well be due to filesystem problems or
>>> to some filesystem filling up (/tmp is a prime suspect) or due to
>>> running out of memory+swap.  Some sort of memory leak sounds pretty
>>> likely actually.
>>>
>>> Probably best to bring the system up in single user mode and run fsck on
>>> all the filesystems manually -- that will show if you've got h/w
>>> problems with drives and possibly with disk controllers or cabling too.
>>>    Then check for overfull filesystems.  You may not find any --
>>> rebooting
>>> can clear a number of conditions where disk space is not released back
>>> to the OS properly after use.  You may or may not find any clues as to
>>> what went wrong in the system logs.  In the absence of any other clues,
>>> the only option is to monitor the server closely and wait for something
>>> similar to happen again.  Hopefully if there is a next time, you'll be
>>> able to catch it and fix the underlying problem before it takes the
>>> machine out a second time.
>>>
>>>
>>>        
>> Yes, I can't log in. I get a login prompt, but no password prompt. I'm
>> going to try ctrl-alt-del and see what happens.
>>
>> Crossing fingers...
>>      
> Sorry  I missed that you can't login.
> Good luck
>    
So, ctrl-alt-del did the trick. I was able to log in and actually, the 
whole box came up and everything seems to be working.

I thought for sure I'd find that my /var directory was full up, but it's 
only at 77% (that's the weak spot on this box... I wish I'd made the 
/var partition bigger.)

The message log is full of these messages:

38054 Jul  2 08:13:02 qu kernel: swap_pager_getswapspace(16): failed

If I run swapinfo, I get this:

[master at qu:log]> swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/ar0s1b       2055952      208  2055744     0%

I looked back in the log file to see if there were any clues when the 
problem began and found this:

Jul  2 03:19:25 qu kernel: swap_pager: out of swap space
Jul  2 03:19:26 qu kernel: swap_pager_getswapspace(7): failed
Jul  2 03:19:26 qu kernel: pid 93543 (mysqld), uid 88, was killed: out 
of swap space
Jul  2 03:19:26 qu kernel: pid 85077 (ruby18), uid 1023, was killed: out 
of swap space
Jul  2 03:19:25 qu root: Check for bad ssh behavior
Jul  2 03:20:05 qu root: Check for bad ssh behavior
Jul  2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed
Jul  2 03:20:49 qu kernel: swap_pager_getswapspace(15): failed
Jul  2 03:20:49 qu kernel: swap_pager_getswapspace(14): failed
Jul  2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed
Jul  2 03:20:49 qu last message repeated 2 times

It pretty much goes on forever from there.

Is there any other place I can look for clues as to why I ran out of 
swap space? This machine is basically a webserver, running apache/mysql 
and ruby on rails. It's been running for over a year with no problems. 
No new software introduced on the box, recently.

-- John