"telinit u" under upstart (upstart's Restart command) with systemd-sysv installed runs systemd

[Timo Aaltonen]

I've today had two machines kill my X session during dist-upgrade, and it happened during libc6 postinst. Dmesg shows that systemd got somehow started during the earlier phase of the upgrade. X logs show a failure with "(EE) Cannot establish any listening sockets" etc..

Adam can fill in the more detailed analysis of what's happening there.. but I think it should be fairly easy to reproduce by installing vivid from an earlier image and dist-upgrading to current, if needed.

Reproducer:
 - Start with a standard utopic VM (e. g. adt-buildvm-ubuntu-cloud -r utopic)
 - Switch apt sources to vivid
 - sudo apt install systemd-sysv
 - sudo telinit u # systemd command, forwards to upstart's /lib/sysvinit/telinit
 - Or directly: sudo /lib/sysvinit/telinit u

[Adam Conrad (adconrad)]

Fairly sure, without actually reproducing myself, that "telinit u" is starting a fresh systemd and taking over PID1, instead of either calling back to upstart (the current PID1) or doing nothing, either of which would be preferable.

[Martin Pitt (pitti)]

Debugging notes:

 - Running telinit u on a current vivid system under upstart, but with systemd-sysv installed (i. e. select upstart from grub, using init=/sbin/upstart) is returning immediately. strace shows that it's connecting to the upstart socket. dmesg shows that upstart-{udev,file}-bridge restarts and otherwise not much is going on.

 - Running systemctl daemon-reexec on that system fails immediately with "Failed to get D-Bus connection: Operation not permitted" as the systemd D-Bus socket isn't available.

[Martin Pitt (pitti)]

 - Reconfiguring glibc on that system does run telinit u, but doesn't hang.

 - I tested/fixed upgrades from trusty and utopic in schroot a few weeks ago, but libc doesn't do the telinit u thing in chroots, so I missed that.

 - I can reproduce that upgrading utopic to vivid in a VM ends up with running systemd. That's clearly not intended.

[Martin Pitt (pitti)]

Stracing "telinit u" shows some interesting actions:

lstat("/run/systemd/system/", 0x7fff0ff73700) = -1 ENOENT (No such file or directory)

→ OK, that's the check whether systemd is running, it's expected to fail under upstart

execve("/lib/sysvinit/telinit", ["telinit", "u"], ... ) = 0

→ Hm, that's from upstart-bin, also looks ok

connect(3, {sa_family=AF_LOCAL, sun_path=@"/com/ubuntu/upstart"}, 22) = 0
[...]
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\3\1\0\0\0\0\1\0\0\0P\0\0\0\1\1o\0\23\0\0\0/com/ubuntu/Upstart\0\0\0\0\0\2\1s\0\25\0\0\0com.ubuntu.Upstart0_6\0\0\0\3\1s\0\7\0\0\0Restart\0", 96}, {"", 0}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 96
exit_group(0) = ?
+++ exited with 0 +++

Hm, nothing unusual here.

[Martin Pitt (pitti)]

An "interesting" variant of that is:

  sudo apt install upstart upstart-bin systemd-sysv
  sudo telinit u

While with utopic's upstart telinit returned immediately and systemd was run, with vivid's upstart the telinit now hangs eternally (but systemd still gets run)

[Martin Pitt (pitti)]

That's because perform_reexec() just reexec's argv[0], which used to be /sbin/init. But that just changed pointing from upstart to pointing to systemd. So upstart's function isn't actually conceptually wrong, but perhaps we could run /sbin/upstart instead of argv[0] (which is likely /sbin/init)?

[Martin Pitt (pitti)]

So we either change upstart's perform_reexec() to run /sbin/upstart, or we change systemd's telinit to not forward an "u" request to upstart, and simply ignore the reload. The latter would break the intention of glibc's restart though (but it's probably not a big deal).

[Adam Conrad (adconrad)]

Fixing upstart to do the right thing is probably good anyway, but since one can't go back in time and fix it, that won't solve the upgrade issue. The only way to solve the upgrade issue is to make systemd not forward the request, I imagine. It could potentially make the first umount on reboot after the upgrade unclean, but I can't think of a better way around this.

[Martin Pitt (pitti)]

Adam points out that we would need to fix that in trusty's and utopic's upstart as well, so let's just work around that in systemd's telinit.

[Martin Pitt (pitti)]

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu&id=0a5d8124369

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-4ubuntu5

---------------
systemd (219-4ubuntu5) vivid; urgency=medium

  * Merge from experimental branch:
    - /lib/lsb/init-functions.d/40-systemd: Don't ignore systemd unit
      dependencies in "degraded" mode. (LP: #1429734)
    - rules-ubuntu/71-power-switch-proliant.rules: Add support for HP ProLiant
      m400 Server Cartridge soft powerdown on Linux 3.16. (LP: #1428811)
  * systemctl: Don't forward telinit u to upstart. This works around upstart's
    Restart() always reexec'ing /sbin/init on Restart(), even if that changes
    to point to systemd during the upgrade. This avoids running systemd during
    a dist-upgrade. (LP: #1430479)
 -- Martin Pitt <email address hidden> Wed, 11 Mar 2015 11:01:36 +0100

[Martin Pitt (pitti)]

As we have a workaround in systemd now, I untag this to clean up the bug list.

[Timo Aaltonen (tjaalton)]

I've verified that the upgrade works now.