Landscape: Upgrade 14.04.5 to 16.04.2 fails unable to reboot

[Damiön la Bagh]

https://github.com/systemd/systemd/pull/10061

[Impact]

 * When logind is not available, shutdown command fails to schedule a shutdown, and despite its intentions to immediately shutdown, does not do so.

[Test Case]

  sudo systemctl mask systemd-logind.service
  sudo systemctl stop systemd-logind.service
  shutdown +1

The expectation is that system goes to shutdown.

It is buggy if the system remains up - i.e. command returns to shell with exit code 1.

[Regression Potential]

 * It is a corner case to run against systemd-shim logind / or logind not otherwise available. The function still performs a clean-shutdown, and should not cause loss of work.

[Other Info]

 * Original bug report, running against systemd-shim/systemd-service post trusty->xenial upgrade, pre-reboot.

Used Landscape (Paid Canonical Subscription) to upgrade one of my machines.

Landscape only shows "In Progress" for more than 8 hours now and asked for a reboot of the machine in a second alert.

In the reboot attempt I get the message:
=========================
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist
=========================

Steps to reproduce:
* Fully updated 14.04.5 machine
* Open Landscape
* Choose the machine
* Choose Packages
* This computer can be upgraded to a newer release
* Apply
* Wait 2 hours
* Alert comes in a seperate Landscape message Machine is ready for reboot
* Choose Info... Power
* Deliver to selected computers as soon as possible
* Error message

I found this thread on reddit about this issue maybe the solution can be built into the upgrade script
https://www.reddit.com/r/linuxquestions/comments/4wy3go/trying_to_run_as_user_instance_but_the_system_has/

[Damiön la Bagh (kat-amsterdam)]

Was able to reboot by sshing into the machine and rebooting with error message but it still went through with the reboot.

Landscape is now stuck on In Progress
  In progress Upgrade from trusty to xenial

[Francis Ginther (fginther)]

This does not require landscape-client. It can be reproduced via:

 * Install machine with 14.04.5 and fully upgrade it.
 * Execute 'do-release-upgrade' to upgrade to 16.04.
 * When the upgrade is finished, exit the release upgrade tool (do not allow the tool to reset the system).
 * Manually attempt to reboot the system with "sudo /sbin/shutdown -r"

The machine will not reboot and the shutdown command will exit with:
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist

/var/log/* logs are attached.

[Francis Ginther (fginther)]

Here's the bug report (from ubuntu-bug /sbin/shutdown):

== ApportVersion =================================
2.20.1-0ubuntu2.5

== Architecture =================================
amd64

== Date =================================
Wed Mar 15 21:35:37 2017

== Dependencies =================================
adduser 3.113+nmu3ubuntu4
apt 1.2.19
apt-utils 1.2.19
cgmanager 0.39-2ubuntu5
dbus 1.10.6-1ubuntu3.3
debconf 1.5.58ubuntu1
debconf-i18n 1.5.58ubuntu1
debianutils 4.7
dmsetup 2:1.02.110-1ubuntu10
dpkg 1.18.4ubuntu1.1
gcc-5-base 5.4.0-6ubuntu1~16.04.4
gcc-6-base 6.0.1-0ubuntu1
gnupg 1.4.20-1ubuntu3.1
gpgv 1.4.20-1ubuntu3.1
init-system-helpers 1.29ubuntu4
libacl1 2.2.52-3
libapparmor1 2.10.95-0ubuntu2.5
libapt-inst2.0 1.2.19
libapt-pkg5.0 1.2.19
libattr1 1:2.4.47-2
libaudit-common 1:2.4.5-1ubuntu2
libaudit1 1:2.4.5-1ubuntu2
libblkid1 2.27.1-6ubuntu3.2
libbz2-1.0 1.0.6-8
libc6 2.23-0ubuntu5
libcap-ng0 0.7.7-1
libcap2 1:2.24-12
libcap2-bin 1:2.24-12
libcgmanager0 0.39-2ubuntu5
libcryptsetup4 2:1.6.6-5ubuntu2
libdb5.3 5.3.28-11
libdbus-1-3 1.10.6-1ubuntu3.3
libdevmapper1.02.1 2:1.02.110-1ubuntu10
libexpat1 2.1.0-7ubuntu0.16.04.2
libfdisk1 2.27.1-6ubuntu3.2
libffi6 3.2.1-4
libgcc1 1:6.0.1-0ubuntu1
libgcrypt20 1.6.5-2ubuntu0.2
libglib2.0-0 2.48.2-0ubuntu1
libglib2.0-data 2.48.2-0ubuntu1
libgpg-error0 1.21-2ubuntu1
libgpm2 1.20.4-6.1
libicu55 55.1-7ubuntu0.1
libkmod2 22-1ubuntu4 [modified: usr/share/doc/libkmod2/changelog.Debian.gz]
liblocale-gettext-perl 1.07-1build1
liblz4-1 0.0~r131-2ubuntu2
liblzma5 5.1.1alpha+20120614-2ubuntu2
libmount1 2.27.1-6ubuntu3.2
libncursesw5 6.0+20160213-1ubuntu1
libnih-dbus1 1.0.3-4.3ubuntu1
libnih1 1.0.3-4.3ubuntu1
libpam-cap 1:2.24-12
libpam-modules 1.1.8-3.2ubuntu2
libpam-modules-bin 1.1.8-3.2ubuntu2
libpam-runtime 1.1.8-3.2ubuntu2
libpam-systemd 229-4ubuntu16
libpam0g 1.1.8-3.2ubuntu2
libpcre3 2:8.38-3.1
libreadline6 6.3-8ubuntu2
libseccomp2 2.2.3-3ubuntu3
libselinux1 2.4-3build2
libsemanage-common 2.3-1build3
libsemanage1 2.3-1build3
libsepol1 2.4-2
libsmartcols1 2.27.1-6ubuntu3.2
libstdc++6 5.4.0-6ubuntu1~16.04.4
libsystemd0 229-4ubuntu16
libtext-charwidth-perl 0.04-7build5
libtext-iconv-perl 1.7-5build4
libtext-wrapi18n-perl 0.06-7.1
libtinfo5 6.0+20160213-1ubuntu1
libudev1 229-4ubuntu16
libusb-0.1-4 2:0.1.12-28
libustr-1.0-1 1.0.4-5
libuuid1 2.27.1-6ubuntu3.2
libxml2 2.9.3+dfsg1-1ubuntu0.1
lsb-base 9.20160110ubuntu0.2
mount 2.27.1-6ubuntu3.2
multiarch-support 2.23-0ubuntu5
passwd 1:4.2-3.1ubuntu5
perl-base 5.22.1-9
readline-common 6.3-8ubuntu2
sed 4.2.2-7
sensible-utils 0.0.9
sgml-base 1.26+nmu4ubuntu1
shared-mime-info 1.5-2ubuntu0.1
systemd 229-4ubuntu16
systemd-shim 9-1bzr4ubuntu1
sysvinit-utils 2.88dsf-59.3ubuntu2
tar 1.28-2.1ubuntu0.1
ubuntu-keyring 2012.05.19
update-motd 3.6-0ubuntu1
util-linux 2.27.1-6ubuntu3.2
uuid-runtime 2.27.1-6ubuntu3.2
xdg-user-dirs 0.15-2ubuntu6
xml-core 0.13+nmu2
zlib1g 1:1.2.8.dfsg-2ubuntu4

== DistroRelease =================================
Ubuntu 16.04

== Package =================================
systemd-sysv 229-4ubuntu16

== PackageArchitecture =================================
amd64

== ProblemType =================================
Bug

== ProcEnviron ===========================...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Remove zesty task; added xenial task.

[Dimitri John Ledkov (xnox)]

Hello, I've looked more into this.

Upstream does admit that shutdown/reboot commands via systemd may fail. And the recommendation there is to progressively use bigger hammers.

See: https://github.com/systemd/systemd/issues/3282

So i believe you should be trying to:

$ systemctl poweroff
$ systemctl -f poweroff
$ systemctl -ff poweroff

(or reboot, or whatever the comamnd you need/want)

Specifically, with upgrade from upstart to systemd, and re-exec, we may not shutdown cleanly, and that should be ok.

[Dimitri John Ledkov (xnox)]

Ping, has above been implemented in landscape? to attempt poweroff, then -f poweroff, then -ff poweroff?

[David Britton (davidpbritton)]

@xnox: not as of yet, no. it's still in our backlog.

[Damiön la Bagh (kat-amsterdam)]

Could we please use

$ systemctl reboot
$ systemctl -f reboot
$ systemctl -ff reboot

instead of poweroff.

My machines are spread all around the country I live in. I don't want to have to drive hundreds of kilometers to go around to turn on all my machines by hand after the upgrade.

Thanks!

[David Britton (davidpbritton)]

For landscape, we will try systemctl reboot, and systemctl -f reboot, this warning in the manpage:

Warning: specifying --force twice with any of these operations might result
           in data loss.

seems like something we don't want to do with managed machines.

[Dave Jones (waveform)]

Seems like there's two issues here which I'll address in separate PRs to the landscape-client:

First is the issue that the landscape-client fails to report the completion (or failure) of the release upgrade process. This turns out to be due to a delayed import in the version of twisted distributed with trusty. Because the import (of twisted.internet.unix) is delayed, by the time we *do* import it (implicitly by calling another twisted method), the twisted version on disk has been replaced with the xenial version and things (unsurprisingly) break. The trivial fix is to pre-emptively import the module prior to starting the release upgrade; I'll push a PR for this in a moment.

The second issue is that landscape-client fails to reboot (or shutdown) the machine immediately following the release upgrade procedure. This is due to the fact landscape-client currently relies upon /sbin/shutdown to handle reboot or shutdown requests. This utility operates happily in its upstart variant (in trusty), and systemd variant (in xenial and beyond), but fails in the post-trusty-to-xenial upgrade environment when the system on disk is effectively xenial (so /sbin/shutdown is a link to systemctl), but the running environment is still partially trusty. It transpires that systemctl (without the double --force --force option) requires both systemd and dbus to be operational in order to successfully (and cleanly) shutdown or restart.

The fix for this is a little more complex as the landscape-client relies upon the scheduling function of /sbin/shutdown to ensure it can request a shutdown and still report back to the server before the machine actually goes down. I'm still investigating the best method which will work reliably in all potential environments.

[Dave Jones (waveform)]

I've now pushed a PR which should alleviate the reboot issue *partially*. The complexity in fixing this arises from the differing behaviours of implementations of the shutdown, poweroff, and reboot commands; upstart's poweroff & reboot commands for example, immediately return (briefly) permitting the caller to query their exit codes. In contrast, systemd's don't return at all unless the poweroff or reboot fails. None of the reboot or poweroff implementations provide a scheduling facility like the /sbin/shutdown implementation (unsurprisingly), but systemd's poweroff and reboot commands *do* work even in the post-trusty upgrade environment (even though systemd's shutdown command doesn't).

The quandry is as follows: we can reliably detect when the shutdown command fails (it exits with an exit code and some error messages), and when it succeeds ... sort of. We currently schedule the shutdown for 4 minutes time and if it hasn't died within 10 seconds we assume it's going to work; this assumes that the implementation won't fail at the end of the schedule but so far that seems a reliable assumption under both upstart and systemd. We cannot *reliably* detect when the poweroff or reboot commands either succeed or fail, but they do seem reliable in practice even in aberrant environments like the post-trusty upgrade.

So, the method I've chosen is as follows: treat shutdown and restart requests as we presently do (schedule /bin/shutdown for 4 minutes time, monitor for 10 seconds). If it succeeds, nothing changes. If it fails, report the failure with an extra message indicating we are going to attempt to force the procedure, then run /sbin/poweroff or /sbin/reboot as appropriate. The result is that in the post-trusty environment (testing on containers and VMs) a reboot request does succeed, but is still reported as failing (albeit with an extended message explaining we're going to try and force it).

We *could* report it as succeeding, but frankly there's no guarantees so it's a choice between giving the user bad news ("sorry your reboot request failed ...") with a possible silver lining ("... but we're going to try another way which *might* work, however we won't be able to tell"), or giving the user good news ("your reboot request worked after we forced it") and hoping we're not lying! The former sounds like the more sensible approach to me, and hence is what is in the PR (https://github.com/CanonicalLtd/landscape-client/pull/55).

[Dimitri John Ledkov (xnox)]

In general, scheduling with shutdown command should work. E.g. tested in lxd container that `$ sudo shutdown +4` works correctly.

I wonder if landscape is affected by:
"""
If the time argument is used, 5 minutes before the system goes down the
/run/nologin file is created to ensure that further logins shall not be
allowed.
"""

Also, I see that 'halt' is used, instead of 'poweroff' in landscape, why is that? I would have thought 'poweroff' is more appropriate (shutdown -P +4, or just shutdown +4)

[Dave Jones (waveform)]

@xnox Scheduling with /sbin/shutdown normally does work ... however, what we're dealing with here is the anomalous situation of an instance which starts off as trusty and is upgraded to xenial. During the upgrade process, the /sbin/shutdown implementation is changed from the upstart one to the systemd one. At the end of the process, /sbin/shutdown is (unsurprisingly) a link to systemctl ... but it doesn't work because it's expecting to be able to talk to systemd which isn't fully operational until a reboot. As Landscape is expecting to be able to reboot via /sbin/shutdown the user relying on Landscape in such a scenario finds themselves in a bit of a bind (having to have some other means of dealing with the server, like SSH).

However, /sbin/reboot and /sbin/poweroff *do* work in this (admittedly rare) scenario, hence the patches committed for this issue leave the system using /sbin/shutdown by default but if/when it fails, they fall back to trying /sbin/reboot or /sbin/poweroff (as appropriate) instead.

As to why -h is used instead of -P, in systemd's implementation of /sbin/shutdown they're exactly the same thing. In the upstart implementation it's more vague; quoting from the man-page: "Requests that the system be either halted or powered off after it has been brought down, with the choice as to which left up to the system". I'm not sure what that means in practice though!

[Dimitri John Ledkov (xnox)]

/sbin/shutdown, post upgrade not working, imho is a severe bug. I'll investigate that. I suspect that because we cannot restart dbus things don't show up there, but /sbin/shutdown should be smart enough to do something. I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

Re differences between halt & poweroff, in systemd, eventually it results in:

case ACTION_HALT:
    (void) reboot(RB_HALT_SYSTEM);

case ACTION_POWEROFF:
    (void) reboot(RB_POWER_OFF);

For completeness reboot typically is:
(void) reboot(RB_AUTOBOOT);

of if parameter was specified (aka androidish - reboot recovery):
(void) raw_reboot(LINUX_REBOOT_CMD_RESTART2, parameter);

This is the same as with upstart's implementation:

case REBOOT:
    reboot (RB_AUTOBOOT);

case HALT:
    reboot (RB_HALT_SYSTEM);

case POWEROFF:
    reboot (RB_POWER_OFF);

case REBOOTCOMMAND:
    syscall (SYS_reboot,
       LINUX_REBOOT_MAGIC1,
       LINUX_REBOOT_MAGIC2,
       LINUX_REBOOT_CMD_RESTART2,
       rebootcommand);

Differences between RB_HALT_SYSTEM and RB_POWER_OFF can be read in the linux manpage here: http://manpages.ubuntu.com/manpages/bionic/en/man2/reboot.2.html

I have not read kernel code in a while for this, but if i'm not mistaken poweroff fallsback to halt, in the kernel, if there is no hardware support to kill power.

[Dave Jones (waveform)]

> /sbin/shutdown, post upgrade not working, imho is a severe bug

I suspect no-one's bothered to report it because as soon as the machine is (somehow) rebooted, the issue goes away (and reproduction then involves the pain of going through a trusty install + xenial upgrade cycle). Still, I'm not entirely convinced this *is* a bug; I'll come back to this in a bit below...

> I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

I thought I'd tested this and concluded it didn't, but it turns out I just didn't wait long enough. When trying "shutdown -r now" there's a fair delay, then it complains about a connection time out, exits with an error, and the system reboots anyway!

    ubuntu@client1:~$ sudo shutdown -r now
    Failed to start reboot.target: Connection timed out
    See system logs and 'systemctl status reboot.target' for details.
    ubuntu@client1:~$ packet_write_wait: Connection to 10.123.236.170 port 22: Broken pipe

So, it is mostly likely the scheduling portion that's failing, and if given an immediate request the operation does still work. I haven't got time to test the -h switch as well, but I'd assume it's probably a similar story.

Now consider those two cases from the perspective of a human administrator:

If you've typed "shutdown -r now" you're expecting an immediate reboot. Even if some bits of the shutdown command fail, you still expect it to do what you asked: reboot immediately. However, if you've just typed "shutdown -r +5" you're expecting a 5 minute delay before reboot. If it can't schedule that delay would you rather it exit with an error and tell you, or just immediately reboot the system? Probably the former given you're not expecting an immediate reboot.

The issue in this case was that it isn't a human scheduling the reboot (directly) and the only reason landscape uses the schedule at all is to determine whether the shutdown command has (likely) worked. Which is a bit of an abuse of the shutdown command's scheduling facility, but I can't think of a better way off the top of my head (that'll reliably work in all implementations).

Anyway, in landscape's case we *would* rather the system immediately reboots when shutdown fails because the delay's only there for the Landscape client's benefit, and we can't guarantee the administrator has any other means of accessing the system for the purpose of rebooting it. So in a sense we're dealing with an interface mismatch: shutdown is (understandably) written for the (direct) use of a human administrator, and landscape is (ab)using it to provide reboot/poweroff facilities on (indirect) behalf of a human administrator.

So we come back to "I can't think of a better way of doing it" (that'll reliably work in all our aforementioned scenarios of upstart, systemd, and post-upgrade-almost-sort-of-systemd).

> Re differences between halt & poweroff, in systemd, eventually it results in:
> [snip]

Yes, but have a look at the arg parsing earlier on (https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c#L8069):

    case 'h':
            if (arg_action != ACTION_HALT)
                    arg_action = ACTION_POWEROFF;
            break;

In ...

[Dimitri John Ledkov (xnox)]

Note to self, I wonder if:
https://github.com/systemd/systemd/commit/c68867da320de296917820e5a76e902a8a86be5a
https://github.com/systemd/systemd/commit/7f96539d45028650f2ba9452095473a9c455d84b

are related.

[Dimitri John Ledkov (xnox)]

Heh, I believe there are historical options re -H -h. As a wild guess, i looked at the sysvinit implementation in Debian (we have removed the package in Ubuntu) it has this:

-h Halt or power off after shutdown.
-H Modifier to the -h flag. Halt action is to halt or drop into boot monitor on sys‐
       tems that support it. Must be used with the -h flag.

So clear as mud, and i guess the "portable" way is to call "shutdown -h -H" or something.

RE: scheduling

I guess you could inspect the system from landscape as follows:
- check system has re-execed into systemd init ( [ -d /run/systemd/system ] )
- check that upstart is gone ( UPSTART_SESSION= initctl version )
- poke the dbus interface to see if logind is there and has support for timing things (probably using busctl http://manpages.ubuntu.com/manpages/xenial/en/man1/busctl.1.html)
- realise the scheduling will not work
- request shutdown now

Imho, failing to schedule thing to shutdown, should shut the machine down instantly. Or i should upgrade machine from trusty to xenial; and like snapshot it; and figure out why scheduling does not work / logind is not there.

[Dimitri John Ledkov (xnox)]

Reading modern systemctl code, if the scheduling fails, the intention is to immediately shutdown.

[Dimitri John Ledkov (xnox)]

https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c#L8548

I wonder if fake systemd-services logind shim is on the dbus, accepts ScheduleShutdown and then does nothing.

[Dimitri John Ledkov (xnox)]

Yeah, this is all still broken upstream. If dbus call to schedule shutdown, fails, immediate shutdown is not actually in fact attempted.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu8

---------------
systemd (239-7ubuntu8) cosmic; urgency=medium

  [ Dimitri John Ledkov ]
  * Cherrypick many bugfixes from master.
  * systemctl: correctly proceed to immediate shutdown if scheduling fails
    (LP: #1670291)

  [ Julian Andres Klode ]
  * Improve networkd states documentation.

 -- Dimitri John Ledkov <email address hidden> Wed, 12 Sep 2018 16:03:08 +0100

[Damiön la Bagh (kat-amsterdam)]

I have still have two machines that need upgrading from 14.04 -> 16.04 -> 18.04. I want to replace them and test the upgrade in the lab.
The upgrade from 16.04->18.04 also fails. I'll report it in a new ticket.

[Robie Basak (racb)]

Hello Damiön, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!