How did I Break ssh?

[Martin McCormick]

	My thanks to all who have offered suggestions as to what
to try.  Here is what I have learned today.

	I can completely remove .ssh from my home directory as in
rm -r ~/.ssh and I get the "host key verification failed."
message rather than an attempt to add a new key to whatever
system I am trying to access.  ssh does recreate .ssh, but it is
empty.

	This is definitely related to my overlaying of the tar
archive as I can demonstrate it on two different systems.  I
simply had not noticed it on the first one I built until now.

	I can use ssh-keygen to generate all my local keys with
no effect except that the keys are good.  If I copy the public
key in to the authorized_keys file on a remote system, it gets me
in to the sick system without a password.  All in-bound
connections work exactly as they should.  No outbound connections
using ssh work at all.

	The system I built that became the source of the tar
balls which almost have built the other two systems couldn't be
better.  Its ssh outbound connections work perfectly.

	This has got to be something that either does not survive
the tar extraction or it is something that only fits the system
it was generated on.  The only files I know about that are unique
are all the keys in /etc/ssh and all the keys in each user home
directory.

	The problem is system-wide on all the effected systems.
I did notice on the other system I cloned that the presence of a
known_hosts file caused any ssh attempt to return the same error
that one gets when there have been too many retries at logging in
to a remote host.

	The verification failure always occurs after the
communication starts and keys are exchanged.

	If I try ssh -v 127.0.0.1 or ssh -v someremotehost.org,
the debug output is almost identical between a working system and
these sick ones except that I am offered a chance to add
127.0.0.1 to the list on the good system while the bad one just
fails.

	If I answer no to the good system, I get the "host key
verification failed" message, also.

	Any other ideas are appreciated.  The idea of building
new systems partly from tar balls appears to mostly work well if
the systems are the same architecture which these are so it is
important to know what is happening here because I suspect it
isn't too hard to fix.

Martin McCormick