Wrong kill mode on shutdown

[Olli Niemi]

I started to use mosh with Ubuntu 15.04 beta2. The packages are from today and updated.
What happens with mosh is that I get "Mosh: You have a detached Mosh session on this server (mosh [<pid>])" from the remote host after shutting down my computer and using mosh again with the same remote host.

I can create this message only if I use "kill -9 <mosh-client-pid>" in the client host. If I use "kill <mosh-client-pid>", I never get this error on the remote host.

I also do not get this error from the remote host if I reboot the client host. I suspect in this case the kill mode is correct.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1446982/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Olli Niemi (olliniem)]

Just guessing, but let's select systemd.

[Martin Pitt (pitti)]

mosh is not a system service and does not have any init.d/service/upstart jobs. Whatever mosh does or does wrong, it does by itself.

[Keith Winstein (keithw)]

Hello,

I think this is working as intended.

When you "kill -9", that is sending the SIGKILL signal -- you are asking the program to shut down *uncleanly* and immediately. The mosh-client dies immediately, and it doesn't have a chance to let the server know that it won't be coming back. The server has no way to distinguish this from a case where the client simply lost its network connection (but might regain it later).

Please use a nicer signal to kill the mosh-client. (E.g. the default for "kill", or just exit from within the client.) Then it will have a chance to shut down the connection cleanly.

(You might still have the problem if you exit the client while it doesn't have a network connection -- then it again doesn't have a chance to let the server know that it's quitting. But that is a different problem that is hard to escape given mosh's design of everlasting, roamable connections.)

[Olli Niemi (olliniem)]

My "kill -9" was a description of similar end result as to what happens on shutdown. Mosh behaves the same way.
I'm not normally using "kill" directly, I just wish it behaves the same way on reboot and shutdown (i.e. normal kill).

I don't even know is there "kill" (SIGTERM) done to mosh on shutdown. This was just a guess.
Another option is that it's missing.

I read somewhere that at least BSD /sbin/shutdown tries to first kill with SIGTERM and after that with SIGKILL.
I don't actually know what the behavior should be on Ubuntu 15.04.

[Keith Winstein (keithw)]

Do you see different behavior on Ubuntu 14.10?

[Olli Niemi (olliniem)]

Just tried mosh from 14.10 -> 15.04. This works correctly.

[Keith Winstein (keithw)]

Can you be a lot more explicit? To make sure I understand you correctly, you are saying that:

(1) With a client running Ubuntu 15.04, shutting down the system does not cleanly close your Mosh connections (it leaves a mosh-server running), but rebooting the system *does* cleanly close your Mosh connections.

(2) With a client running Ubuntu 14.10 but nothing else changed, BOTH shutting down AND rebooting the client cleanly close your Mosh connections.

Is this correct?

[Olli Niemi (olliniem)]

(1) I am connecting from a machine running Ubuntu release 15.04 (mosh version 1.2.4a) to one Raspberry Pi machine running Raspbian Wheezy release (mosh version 1.2.3). After shutdown and restart of the machine running Ubuntu release 15.04 (mosh version 1.2.4a) I get this error from the Raspberry Pi machine after a reconnect with mosh from the same 15.04 machine (mosh version 1.2.4a). After restart I do *not* get the this error from the Raspbian machine (mosh version 1.2.3) when doing a reconnect with mosh (version 1.2.4a).

(2) Connecting from the 14.10 machine (mosh version 1.2.4a) to the 15.04 machine (mosh version 1.2.4a). I did just a quick test with the 14.10 machine: shutdown only case (mosh version 1.2.4a). After shutdown and restart of the 14.10 machine (mosh version 1.2.4a) I do *not* get this error from the 14.10 machine when doing a reconnect with mosh (version version 1.2.4a).

[Keith Winstein (keithw)]

I'm trying to help you out here, but you are making things more confusing than necessary.

If you can document clearly that Ubuntu 15.04 behaves differently from 14.10 in the shutdown sequence (e.g., perhaps Ubuntu 14.10's shutdown sequence sent a SIGTERM *before* cutting off the network, and Ubuntu 15.04 cuts off the network first), you might be able to get the systemd people interested again.

But you have to document it very clearly and change only one thing at a time.

[Olli Niemi (olliniem)]

Are there any Ubuntu Mosh guys who take the full end-to-end responsibility of Mosh (or systemd) working correctly there?

[Anders Kaseorg (andersk)]

Olli, Mosh is in the universe repository, so that responsibility falls to the community. In practice, the set of people working on Mosh in Ubuntu consists of Keith, who is both the upstream maintainer and the maintainer of the Debian package that Ubuntu automatically imports. He knows what he’s talking about, and you should pay attention to his comments. The process of getting volunteers interested in spending their spare time to fix your bug is not always easy, but Keith has suggested a clear path forward, and the ball is in your court now.

[Olli Niemi (olliniem)]

OK, I will try with strace later, hope it's sufficient proof.

[Olli Niemi (olliniem)]

Unfortunately strace didn't reveal much. However, I created a small program to show the signal.

Using test.c I can see that when testing with Ubuntu 14.10 the program always creates the "testing.txt" file on restart.
However, if I test with Ubuntu 15.04, that file is never created in the same use case.

Contrary to my previous claim, it behaves the same way in reboot & shutdown:
Mosh gets the error on reconnect and "test.c" never creates the testing.txt file.
SIGTERM is never received.

[Keith Winstein (keithw)]

This does seem to be a systemd problem that is affecting a lot of different applications -- basically any application that would like a SIGTERM on reboot/shutdown so that it can save or clean up resources gracefully.

See https://bugzilla.redhat.com/show_bug.cgi?id=1141137 (upstream systemd issue)

https://bugzilla.redhat.com/show_bug.cgi?id=1170765 (systemd issue affecting bash's attempt to save history)

https://github.com/cockpit-project/cockpit/issues/1155 (systemd issue affecting cockpit)

http://lists.freedesktop.org/archives/systemd-devel/2014-October/024452.html (discussion)

My suggestion would be that you open a new bug report against systemd in Ubuntu 15.04 and mention that it is affecting bash and mosh (and probably every other application that would like to clean up gracefully on shutdown or reboot).

[Olli Niemi (olliniem)]

Thanks Keith for the summary. This bug can be closed.

[Olli Niemi (olliniem)]

New bug created: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1448259