Mosh regression between 10.x and 11-stable

[john hood]

On 8/10/16 4:18 AM, Peter Jeremy wrote:
> I recently updated one of my VPS hosts from 10.3-RELEASE-p5 to 11.0-BETA4
> r303811 and mosh to that host from my Linux laptop stopped working.  All
> I get on the laptop is:
> $ mosh remotehost
> Connection to remotehost closed.
> /usr/bin/mosh: Did not find mosh server startup message.
> 
> I've tried rebuilding mosh (and all dependencies) on the host to no avail.

I'm a mosh maintainer.  mosh 1.2.5 (from ports) and mosh master (just
last night tagged as 1.2.6, alas) work fine for me on my two 11.0-BETA4
systems, one local and one remote.

> This isn't the DSA change that's been discussed elsewhere: I can SSH from my
> laptop to the host without problem.  I can also manually invoke mosh-client
> and mosh-server and it works.  Unfortunately, mosh has no provision for
> debugging.  I've tried hacking the mosh perl script to make it more verbose
> and that shows that:
> 1) the "MOSH CONNECT" message isn't making it out of the local ssh process.

Do you know if the message is getting out of mosh-server?  into sshd?
Do you know if mosh-server is actually running?  (It will log utmp
entries on startup.)

Mosh's debugging/logging isn't very good, but 'mosh-server new -v 2>
logfile' does produce some useful info (mostly logging of network traffic).

> 2) it's racy because I can get it from "always fails" to "sometimes works".

How do you get it there?

> My suspicion is that something has changed in either sshd or TCP that
> is resulting in the connection going away before the stdout from the
> remote mosh-server makes it out from the local ssh process.

mosh does 'ssh -t' and uses ptys.  That's another potential point the
message could get dropped.

> I've looked at tcpdump's of both successful and failed SSH sessions
> but don't see anything obviously different (encryption makes it
> difficult to decode the session).
> 
> Has anyone else seen this behaviour or have any ideas what might be
> causing it?

Common suspects include issues with shell login/invocation of mosh (are
you making sure it's reachable in /usr/local/bin with $PATH or
'--server=/usr/local/bin/mosh'?  are your login shell and its login
scripts unusual?)

On Linux we've had issues with ecryptfs and systemd breaking mosh-server
when the ssh session ends, but I don't think that applies here.

regards,

  --jh