Mosh regression between 10.x and 11-stable

[john hood]

On 8/13/16 4:30 AM, Peter Jeremy wrote:
> Hi John,
> 
> Sorry, I got side-tracked.
> 
> On 2016-Aug-12 16:37:15 -0400, John Hood <cgull at glup.org> wrote:
>> >Could I ask you to look at this a little further?  On the one hand, it
>> >sure looks like a Mosh issue, and tcdrain() solves it-- but on the other
>> >hand, this is a regression on FreeBSD and we don't see this issue on any
>> >other OS.  I'd like to fully understand this and make sure that this is
>> >not a kernel issue.
> It's got me puzzled as well.  And it's only getting wierder...  The
> following is using an unmodified mosh-1.2.5, built from the port, as
> the server on FreeBSD 11.0-BETA4 r303957.  The client is 1.2.4a-1ubuntu1
> on Linux.  the standard driver script consistently fails but adding a
> "print" makes it mostly work.  Where there's a "[mosh is exiting.]"
> message, it was successful (and would report that there were other
> orphaned servers since I wasn't waiting the 60 seconds for servers to
> die between invocations).  For completeness, I've tried ktrace'ing
> mosh-server but can't make it fail when I do so.

I've now managed to reproduce the issue *and* ktrace it (and sshd) on a
VPS, with a single-CPU VM on a badly-oversubscribed VMWare host and OS X
client.

mosh-server shows a straightforward execution trace, the parent
successfully writes the MOSH CONNECT message on stdout, forks and exits.
 About a millisecond later, the child starts running, writes the verbose
copyright/etc. message on stderr, and gets EIO (and ignores it).
Shortly thereafter it closes its pty slave fds on stdin/stdout/stderr.
It continues normally from there.

In that millisecond, the sshd trace shows that it catches SIGCHLD,
writes utmp info, wait()s for a child and gets mosh-server's pid, and
does a final read() that returns 0 bytes.  It then closes the pty
master, without doing revoke() or any ioctls.

So the pty driver is clearly dropping the MOSH CONNECT message.  It's a
lot less clear whether that's a bug.  On the one hand,
<http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html#tag_11_01_11>
states that output should be drained on final close (which in this case
is done by the forked mosh-server).  On the other hand, the login shell
is session leader, and
<http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html#tag_11_01_03>
requires that the terminal be disassociated from the session when it exits.

I suspect that this problem is most visible on single-core machines,
because on a multi-core machine the pty driver, mosh-servers, and sshd
will run with different ordering, and I suspect that the mosh-server
child gets to close the pty slave before sshd closes the pty master.

I wrote a Xenix serial driver ages ago and I can see arguments for
either draining or dropping final output.  And Mosh's behavior is a
little questionable here.  So I'm not sure whether to call this a kernel
pty bug or not.  But the workaround in Mosh is easy enough, tcdrain(),
so I'm doing that anyway.

regards,

  --jh