inbound ssh ceased on 4 servers at same time

[Michal Mertl]

John Brooks wrote:
> sshd is running on the affected machines
> 
> no errors on console or logs, just times out waiting for
> the password prompt. interestingly: when investigating this
> at the console, attempting ssh sessions from the db server
> and backup server to the file server (these two are 'deeper'
> in the network so there was never an occasion to ssh FROM
> them before) produced the std warning about an unknown host
> prompting for inclusion in the ~/.ssh/known_hosts file.
> 
> dns is not really involved, the ssh session is sent to the
> ip address directly as in "ssh john at 10.3.3.10"

With SSH a host name lookup is always involved. The server performs
reverse and forward lookup on the connecting IP. I've recently also had
problem with SSH and it was DNS issue.

A good test if a daemon is running is connecting with telnet to its port
- you can see three kinds of responses - connection reset (refused) on a
closed port (nothing listens on this port), nothing (just message
'Trying...' - on non-existent/firewalled host/port) and established
connection (telnet says 'Connected to...'). If you get 'Connected' with
ssh and nothing is displayed it's in 99% of cases a DNS issue. If you
get some message or the server just disconnects you (you get 'Connection
closed' message) it might be tcpwrappers' decision (man 5 hosts_access).

If you happen to have intermittent DNS issues you might better put the
important hosts into /etc/hosts which is by default queried before the
DNS servers.

The other daemons working need not to mean much - they may not do a
reverse DNS lookup on connecting IP.

> ping works in both directions as does all other network
> services (internal mysql, intranet http, pop3, smtp, smbd, 
> nmdb, dns). network hardware and cabling issues have been 
> effectively ruled out.
> 
> --
> John Brooks
> john at day-light.com 
> 
> > -----Original Message-----
> > From: Glenn Dawson [mailto:glenn at antimatter.net]
> > Sent: Saturday, June 04, 2005 2:56 PM
> > To: john at day-light.com
> > Cc: freebsd-questions at freebsd.org
> > Subject: Re: inbound ssh ceased on 4 servers at same time
> > 
> > 
> > At 09:05 AM 6/4/2005, you wrote:
> > >Yesterday at about noon, all four freebsd servers on a clients lan
> > >quit accepting ssh connections. All were running 4.11-release-p4,
> > >and had been cvsup'd at the same time from cvs-10, cvs-11, or
> > >cvs-12. Outbound ssh (from console of the affected boxes) works as
> > >expected, both to local openbsd boxes and to remote locations.
> > >There are no host based firewalls involved, and all other network
> > >services are operating correctly. Netstat shows port 22 as listening.
> > >At 11:20 am (40 minutes earlier), ssh was working properly on all boxes.
> > >
> > >Has anybody encountered a situation like this before?
> > 
> > Not specifically, but the first things I would check:
> > 
> > is sshd running on the affected machines?
> > 
> > when trying to connect to the affected machines, do the clients give any 
> > error messages?  or does the connection just time out?
> > 
> > are there any relevant entries in the log files on the affected 
> > machines?  specifically /var/log/messages and /var/log/auth.log
> > 
> > are the affected machines using the same name server? and if they 
> > are, can 
> > the affected machines do forward and reverse lookups for the IP of the 
> > system you are trying to connect from?
> > 
> > -Glenn
> > 
> > 
> > >--
> > >John Brooks
> > >john at day-light.com
> > >_______________________________________________