Landscape: Upgrade 14.04.5 to 16.04.2 fails unable to reboot

[Damiön la Bagh]

https://github.com/systemd/systemd/pull/10061

[Impact]

 * When logind is not available, shutdown command fails to schedule a shutdown, and despite its intentions to immediately shutdown, does not do so.

[Test Case]

  sudo systemctl mask systemd-logind.service
  sudo systemctl stop systemd-logind.service
  shutdown +1

The expectation is that system goes to shutdown.

It is buggy if the system remains up - i.e. command returns to shell with exit code 1.

[Regression Potential]

 * It is a corner case to run against systemd-shim logind / or logind not otherwise available. The function still performs a clean-shutdown, and should not cause loss of work.

[Other Info]

 * Original bug report, running against systemd-shim/systemd-service post trusty->xenial upgrade, pre-reboot.

Used Landscape (Paid Canonical Subscription) to upgrade one of my machines.

Landscape only shows "In Progress" for more than 8 hours now and asked for a reboot of the machine in a second alert.

In the reboot attempt I get the message:
=========================
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist
=========================

Steps to reproduce:
* Fully updated 14.04.5 machine
* Open Landscape
* Choose the machine
* Choose Packages
* This computer can be upgraded to a newer release
* Apply
* Wait 2 hours
* Alert comes in a seperate Landscape message Machine is ready for reboot
* Choose Info... Power
* Deliver to selected computers as soon as possible
* Error message

I found this thread on reddit about this issue maybe the solution can be built into the upgrade script
https://www.reddit.com/r/linuxquestions/comments/4wy3go/trying_to_run_as_user_instance_but_the_system_has/

[Damiön la Bagh (kat-amsterdam)]

Was able to reboot by sshing into the machine and rebooting with error message but it still went through with the reboot.

Landscape is now stuck on In Progress
  In progress Upgrade from trusty to xenial

[Francis Ginther (fginther)]

This does not require landscape-client. It can be reproduced via:

 * Install machine with 14.04.5 and fully upgrade it.
 * Execute 'do-release-upgrade' to upgrade to 16.04.
 * When the upgrade is finished, exit the release upgrade tool (do not allow the tool to reset the system).
 * Manually attempt to reboot the system with "sudo /sbin/shutdown -r"

The machine will not reboot and the shutdown command will exit with:
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist

/var/log/* logs are attached.

[Francis Ginther (fginther)]

Here's the bug report (from ubuntu-bug /sbin/shutdown):

== ApportVersion =================================
2.20.1-0ubuntu2.5

== Architecture =================================
amd64

== Date =================================
Wed Mar 15 21:35:37 2017

== Dependencies =================================
adduser 3.113+nmu3ubuntu4
apt 1.2.19
apt-utils 1.2.19
cgmanager 0.39-2ubuntu5
dbus 1.10.6-1ubuntu3.3
debconf 1.5.58ubuntu1
debconf-i18n 1.5.58ubuntu1
debianutils 4.7
dmsetup 2:1.02.110-1ubuntu10
dpkg 1.18.4ubuntu1.1
gcc-5-base 5.4.0-6ubuntu1~16.04.4
gcc-6-base 6.0.1-0ubuntu1
gnupg 1.4.20-1ubuntu3.1
gpgv 1.4.20-1ubuntu3.1
init-system-helpers 1.29ubuntu4
libacl1 2.2.52-3
libapparmor1 2.10.95-0ubuntu2.5
libapt-inst2.0 1.2.19
libapt-pkg5.0 1.2.19
libattr1 1:2.4.47-2
libaudit-common 1:2.4.5-1ubuntu2
libaudit1 1:2.4.5-1ubuntu2
libblkid1 2.27.1-6ubuntu3.2
libbz2-1.0 1.0.6-8
libc6 2.23-0ubuntu5
libcap-ng0 0.7.7-1
libcap2 1:2.24-12
libcap2-bin 1:2.24-12
libcgmanager0 0.39-2ubuntu5
libcryptsetup4 2:1.6.6-5ubuntu2
libdb5.3 5.3.28-11
libdbus-1-3 1.10.6-1ubuntu3.3
libdevmapper1.02.1 2:1.02.110-1ubuntu10
libexpat1 2.1.0-7ubuntu0.16.04.2
libfdisk1 2.27.1-6ubuntu3.2
libffi6 3.2.1-4
libgcc1 1:6.0.1-0ubuntu1
libgcrypt20 1.6.5-2ubuntu0.2
libglib2.0-0 2.48.2-0ubuntu1
libglib2.0-data 2.48.2-0ubuntu1
libgpg-error0 1.21-2ubuntu1
libgpm2 1.20.4-6.1
libicu55 55.1-7ubuntu0.1
libkmod2 22-1ubuntu4 [modified: usr/share/doc/libkmod2/changelog.Debian.gz]
liblocale-gettext-perl 1.07-1build1
liblz4-1 0.0~r131-2ubuntu2
liblzma5 5.1.1alpha+20120614-2ubuntu2
libmount1 2.27.1-6ubuntu3.2
libncursesw5 6.0+20160213-1ubuntu1
libnih-dbus1 1.0.3-4.3ubuntu1
libnih1 1.0.3-4.3ubuntu1
libpam-cap 1:2.24-12
libpam-modules 1.1.8-3.2ubuntu2
libpam-modules-bin 1.1.8-3.2ubuntu2
libpam-runtime 1.1.8-3.2ubuntu2
libpam-systemd 229-4ubuntu16
libpam0g 1.1.8-3.2ubuntu2
libpcre3 2:8.38-3.1
libreadline6 6.3-8ubuntu2
libseccomp2 2.2.3-3ubuntu3
libselinux1 2.4-3build2
libsemanage-common 2.3-1build3
libsemanage1 2.3-1build3
libsepol1 2.4-2
libsmartcols1 2.27.1-6ubuntu3.2
libstdc++6 5.4.0-6ubuntu1~16.04.4
libsystemd0 229-4ubuntu16
libtext-charwidth-perl 0.04-7build5
libtext-iconv-perl 1.7-5build4
libtext-wrapi18n-perl 0.06-7.1
libtinfo5 6.0+20160213-1ubuntu1
libudev1 229-4ubuntu16
libusb-0.1-4 2:0.1.12-28
libustr-1.0-1 1.0.4-5
libuuid1 2.27.1-6ubuntu3.2
libxml2 2.9.3+dfsg1-1ubuntu0.1
lsb-base 9.20160110ubuntu0.2
mount 2.27.1-6ubuntu3.2
multiarch-support 2.23-0ubuntu5
passwd 1:4.2-3.1ubuntu5
perl-base 5.22.1-9
readline-common 6.3-8ubuntu2
sed 4.2.2-7
sensible-utils 0.0.9
sgml-base 1.26+nmu4ubuntu1
shared-mime-info 1.5-2ubuntu0.1
systemd 229-4ubuntu16
systemd-shim 9-1bzr4ubuntu1
sysvinit-utils 2.88dsf-59.3ubuntu2
tar 1.28-2.1ubuntu0.1
ubuntu-keyring 2012.05.19
update-motd 3.6-0ubuntu1
util-linux 2.27.1-6ubuntu3.2
uuid-runtime 2.27.1-6ubuntu3.2
xdg-user-dirs 0.15-2ubuntu6
xml-core 0.13+nmu2
zlib1g 1:1.2.8.dfsg-2ubuntu4

== DistroRelease =================================
Ubuntu 16.04

== Package =================================
systemd-sysv 229-4ubuntu16

== PackageArchitecture =================================
amd64

== ProblemType =================================
Bug

== ProcEnviron ===========================...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Remove zesty task; added xenial task.

[Dimitri John Ledkov (xnox)]

Hello, I've looked more into this.

Upstream does admit that shutdown/reboot commands via systemd may fail. And the recommendation there is to progressively use bigger hammers.

See: https://github.com/systemd/systemd/issues/3282

So i believe you should be trying to:

$ systemctl poweroff
$ systemctl -f poweroff
$ systemctl -ff poweroff

(or reboot, or whatever the comamnd you need/want)

Specifically, with upgrade from upstart to systemd, and re-exec, we may not shutdown cleanly, and that should be ok.

[Dimitri John Ledkov (xnox)]

Ping, has above been implemented in landscape? to attempt poweroff, then -f poweroff, then -ff poweroff?

[David Britton (dpb)]

@xnox: not as of yet, no. it's still in our backlog.

[Damiön la Bagh (kat-amsterdam)]

Could we please use

$ systemctl reboot
$ systemctl -f reboot
$ systemctl -ff reboot

instead of poweroff.

My machines are spread all around the country I live in. I don't want to have to drive hundreds of kilometers to go around to turn on all my machines by hand after the upgrade.

Thanks!

[David Britton (dpb)]

For landscape, we will try systemctl reboot, and systemctl -f reboot, this warning in the manpage:

Warning: specifying --force twice with any of these operations might result
           in data loss.

seems like something we don't want to do with managed machines.

[Dave Jones (waveform)]

Seems like there's two issues here which I'll address in separate PRs to the landscape-client:

First is the issue that the landscape-client fails to report the completion (or failure) of the release upgrade process. This turns out to be due to a delayed import in the version of twisted distributed with trusty. Because the import (of twisted.internet.unix) is delayed, by the time we *do* import it (implicitly by calling another twisted method), the twisted version on disk has been replaced with the xenial version and things (unsurprisingly) break. The trivial fix is to pre-emptively import the module prior to starting the release upgrade; I'll push a PR for this in a moment.

The second issue is that landscape-client fails to reboot (or shutdown) the machine immediately following the release upgrade procedure. This is due to the fact landscape-client currently relies upon /sbin/shutdown to handle reboot or shutdown requests. This utility operates happily in its upstart variant (in trusty), and systemd variant (in xenial and beyond), but fails in the post-trusty-to-xenial upgrade environment when the system on disk is effectively xenial (so /sbin/shutdown is a link to systemctl), but the running environment is still partially trusty. It transpires that systemctl (without the double --force --force option) requires both systemd and dbus to be operational in order to successfully (and cleanly) shutdown or restart.

The fix for this is a little more complex as the landscape-client relies upon the scheduling function of /sbin/shutdown to ensure it can request a shutdown and still report back to the server before the machine actually goes down. I'm still investigating the best method which will work reliably in all potential environments.

[Dave Jones (waveform)]

I've now pushed a PR which should alleviate the reboot issue *partially*. The complexity in fixing this arises from the differing behaviours of implementations of the shutdown, poweroff, and reboot commands; upstart's poweroff & reboot commands for example, immediately return (briefly) permitting the caller to query their exit codes. In contrast, systemd's don't return at all unless the poweroff or reboot fails. None of the reboot or poweroff implementations provide a scheduling facility like the /sbin/shutdown implementation (unsurprisingly), but systemd's poweroff and reboot commands *do* work even in the post-trusty upgrade environment (even though systemd's shutdown command doesn't).

The quandry is as follows: we can reliably detect when the shutdown command fails (it exits with an exit code and some error messages), and when it succeeds ... sort of. We currently schedule the shutdown for 4 minutes time and if it hasn't died within 10 seconds we assume it's going to work; this assumes that the implementation won't fail at the end of the schedule but so far that seems a reliable assumption under both upstart and systemd. We cannot *reliably* detect when the poweroff or reboot commands either succeed or fail, but they do seem reliable in practice even in aberrant environments like the post-trusty upgrade.

So, the method I've chosen is as follows: treat shutdown and restart requests as we presently do (schedule /bin/shutdown for 4 minutes time, monitor for 10 seconds). If it succeeds, nothing changes. If it fails, report the failure with an extra message indicating we are going to attempt to force the procedure, then run /sbin/poweroff or /sbin/reboot as appropriate. The result is that in the post-trusty environment (testing on containers and VMs) a reboot request does succeed, but is still reported as failing (albeit with an extended message explaining we're going to try and force it).

We *could* report it as succeeding, but frankly there's no guarantees so it's a choice between giving the user bad news ("sorry your reboot request failed ...") with a possible silver lining ("... but we're going to try another way which *might* work, however we won't be able to tell"), or giving the user good news ("your reboot request worked after we forced it") and hoping we're not lying! The former sounds like the more sensible approach to me, and hence is what is in the PR (https://github.com/CanonicalLtd/landscape-client/pull/55).

[Dimitri John Ledkov (xnox)]

In general, scheduling with shutdown command should work. E.g. tested in lxd container that `$ sudo shutdown +4` works correctly.

I wonder if landscape is affected by:
"""
If the time argument is used, 5 minutes before the system goes down the
/run/nologin file is created to ensure that further logins shall not be
allowed.
"""

Also, I see that 'halt' is used, instead of 'poweroff' in landscape, why is that? I would have thought 'poweroff' is more appropriate (shutdown -P +4, or just shutdown +4)

[Dave Jones (waveform)]

@xnox Scheduling with /sbin/shutdown normally does work ... however, what we're dealing with here is the anomalous situation of an instance which starts off as trusty and is upgraded to xenial. During the upgrade process, the /sbin/shutdown implementation is changed from the upstart one to the systemd one. At the end of the process, /sbin/shutdown is (unsurprisingly) a link to systemctl ... but it doesn't work because it's expecting to be able to talk to systemd which isn't fully operational until a reboot. As Landscape is expecting to be able to reboot via /sbin/shutdown the user relying on Landscape in such a scenario finds themselves in a bit of a bind (having to have some other means of dealing with the server, like SSH).

However, /sbin/reboot and /sbin/poweroff *do* work in this (admittedly rare) scenario, hence the patches committed for this issue leave the system using /sbin/shutdown by default but if/when it fails, they fall back to trying /sbin/reboot or /sbin/poweroff (as appropriate) instead.

As to why -h is used instead of -P, in systemd's implementation of /sbin/shutdown they're exactly the same thing. In the upstart implementation it's more vague; quoting from the man-page: "Requests that the system be either halted or powered off after it has been brought down, with the choice as to which left up to the system". I'm not sure what that means in practice though!

[Dimitri John Ledkov (xnox)]

/sbin/shutdown, post upgrade not working, imho is a severe bug. I'll investigate that. I suspect that because we cannot restart dbus things don't show up there, but /sbin/shutdown should be smart enough to do something. I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

Re differences between halt & poweroff, in systemd, eventually it results in:

case ACTION_HALT:
    (void) reboot(RB_HALT_SYSTEM);

case ACTION_POWEROFF:
    (void) reboot(RB_POWER_OFF);

For completeness reboot typically is:
(void) reboot(RB_AUTOBOOT);

of if parameter was specified (aka androidish - reboot recovery):
(void) raw_reboot(LINUX_REBOOT_CMD_RESTART2, parameter);

This is the same as with upstart's implementation:

case REBOOT:
    reboot (RB_AUTOBOOT);

case HALT:
    reboot (RB_HALT_SYSTEM);

case POWEROFF:
    reboot (RB_POWER_OFF);

case REBOOTCOMMAND:
    syscall (SYS_reboot,
       LINUX_REBOOT_MAGIC1,
       LINUX_REBOOT_MAGIC2,
       LINUX_REBOOT_CMD_RESTART2,
       rebootcommand);

Differences between RB_HALT_SYSTEM and RB_POWER_OFF can be read in the linux manpage here: http://manpages.ubuntu.com/manpages/bionic/en/man2/reboot.2.html

I have not read kernel code in a while for this, but if i'm not mistaken poweroff fallsback to halt, in the kernel, if there is no hardware support to kill power.

[Dave Jones (waveform)]

> /sbin/shutdown, post upgrade not working, imho is a severe bug

I suspect no-one's bothered to report it because as soon as the machine is (somehow) rebooted, the issue goes away (and reproduction then involves the pain of going through a trusty install + xenial upgrade cycle). Still, I'm not entirely convinced this *is* a bug; I'll come back to this in a bit below...

> I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

I thought I'd tested this and concluded it didn't, but it turns out I just didn't wait long enough. When trying "shutdown -r now" there's a fair delay, then it complains about a connection time out, exits with an error, and the system reboots anyway!

    ubuntu@client1:~$ sudo shutdown -r now
    Failed to start reboot.target: Connection timed out
    See system logs and 'systemctl status reboot.target' for details.
    ubuntu@client1:~$ packet_write_wait: Connection to 10.123.236.170 port 22: Broken pipe

So, it is mostly likely the scheduling portion that's failing, and if given an immediate request the operation does still work. I haven't got time to test the -h switch as well, but I'd assume it's probably a similar story.

Now consider those two cases from the perspective of a human administrator:

If you've typed "shutdown -r now" you're expecting an immediate reboot. Even if some bits of the shutdown command fail, you still expect it to do what you asked: reboot immediately. However, if you've just typed "shutdown -r +5" you're expecting a 5 minute delay before reboot. If it can't schedule that delay would you rather it exit with an error and tell you, or just immediately reboot the system? Probably the former given you're not expecting an immediate reboot.

The issue in this case was that it isn't a human scheduling the reboot (directly) and the only reason landscape uses the schedule at all is to determine whether the shutdown command has (likely) worked. Which is a bit of an abuse of the shutdown command's scheduling facility, but I can't think of a better way off the top of my head (that'll reliably work in all implementations).

Anyway, in landscape's case we *would* rather the system immediately reboots when shutdown fails because the delay's only there for the Landscape client's benefit, and we can't guarantee the administrator has any other means of accessing the system for the purpose of rebooting it. So in a sense we're dealing with an interface mismatch: shutdown is (understandably) written for the (direct) use of a human administrator, and landscape is (ab)using it to provide reboot/poweroff facilities on (indirect) behalf of a human administrator.

So we come back to "I can't think of a better way of doing it" (that'll reliably work in all our aforementioned scenarios of upstart, systemd, and post-upgrade-almost-sort-of-systemd).

> Re differences between halt & poweroff, in systemd, eventually it results in:
> [snip]

Yes, but have a look at the arg parsing earlier on (https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c#L8069):

    case 'h':
            if (arg_action != ACTION_HALT)
                    arg_action = ACTION_POWEROFF;
            break;

In ...

[Dimitri John Ledkov (xnox)]

Note to self, I wonder if:
https://github.com/systemd/systemd/commit/c68867da320de296917820e5a76e902a8a86be5a
https://github.com/systemd/systemd/commit/7f96539d45028650f2ba9452095473a9c455d84b

are related.

[Dimitri John Ledkov (xnox)]

Heh, I believe there are historical options re -H -h. As a wild guess, i looked at the sysvinit implementation in Debian (we have removed the package in Ubuntu) it has this:

-h Halt or power off after shutdown.
-H Modifier to the -h flag. Halt action is to halt or drop into boot monitor on sys‐
       tems that support it. Must be used with the -h flag.

So clear as mud, and i guess the "portable" way is to call "shutdown -h -H" or something.

RE: scheduling

I guess you could inspect the system from landscape as follows:
- check system has re-execed into systemd init ( [ -d /run/systemd/system ] )
- check that upstart is gone ( UPSTART_SESSION= initctl version )
- poke the dbus interface to see if logind is there and has support for timing things (probably using busctl http://manpages.ubuntu.com/manpages/xenial/en/man1/busctl.1.html)
- realise the scheduling will not work
- request shutdown now

Imho, failing to schedule thing to shutdown, should shut the machine down instantly. Or i should upgrade machine from trusty to xenial; and like snapshot it; and figure out why scheduling does not work / logind is not there.

[Dimitri John Ledkov (xnox)]

Reading modern systemctl code, if the scheduling fails, the intention is to immediately shutdown.

[Dimitri John Ledkov (xnox)]

https://github.com/systemd/systemd/blob/master/src/systemctl/systemctl.c#L8548

I wonder if fake systemd-services logind shim is on the dbus, accepts ScheduleShutdown and then does nothing.

[Dimitri John Ledkov (xnox)]

Yeah, this is all still broken upstream. If dbus call to schedule shutdown, fails, immediate shutdown is not actually in fact attempted.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu8

---------------
systemd (239-7ubuntu8) cosmic; urgency=medium

  [ Dimitri John Ledkov ]
  * Cherrypick many bugfixes from master.
  * systemctl: correctly proceed to immediate shutdown if scheduling fails
    (LP: #1670291)

  [ Julian Andres Klode ]
  * Improve networkd states documentation.

 -- Dimitri John Ledkov <email address hidden> Wed, 12 Sep 2018 16:03:08 +0100

[Damiön la Bagh (kat-amsterdam)]

I have still have two machines that need upgrading from 14.04 -> 16.04 -> 18.04. I want to replace them and test the upgrade in the lab.
The upgrade from 16.04->18.04 also fails. I'll report it in a new ticket.

[Robie Basak (racb)]

Hello Damiön, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

root@subtle-cobra:~# dpkg-query -W systemd
systemd 229-4ubuntu21.4
root@subtle-cobra:~# systemctl mask systemd-logind
Created symlink from /etc/systemd/system/systemd-logind.service to /dev/null.
root@subtle-cobra:~# systemctl stop systemd-logind
root@subtle-cobra:~# shutdown +1
Failed to set wall message, ignoring: Unit systemd-logind.service is masked.
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Unit systemd-logind.service is masked.
root@subtle-cobra:~# echo $?
1
root@subtle-cobra:~# sleep 60

root@subtle-cobra:~# nano /etc/apt/sources.list
root@subtle-cobra:~# apt update
root@subtle-cobra:~# apt full-upgrade

root@subtle-cobra:~# dpkg-query -W systemd
systemd 229-4ubuntu21.5
root@subtle-cobra:~# shutdown +1
Failed to set wall message, ignoring: Unit systemd-logind.service is masked.
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Unit systemd-logind.service is masked.
root@subtle-cobra:~# xnox@sochi:~$
(container shutdown correctly)

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu21.5

---------------
systemd (229-4ubuntu21.5) xenial; urgency=medium

  [ Dimitri John Ledkov ]
  * systemctl: correctly proceed to immediate shutdown if scheduling fails
    (LP: #1670291)
  * hwdb: update micmute on Dell laptops. (LP: #1738153)
  * hwdb: Use wlan keycode for all Dell systems. (LP: #1762385)
  * units: Disable journald Watchdog (LP: #1773148)

  [ Mauricio Faria de Oliveira ]
  * core: Fix for service to enter the 'failed' state (rather than 'inactive') after it repeatedly fails restart.
    (LP: #1795658)

  [ Dimitri John Ledkov ]
  * Disable dh_installinit generation of tmpfiles for the systemd package.
    (LP: #1748147)

 -- Dimitri John Ledkov <email address hidden> Mon, 08 Oct 2018 16:10:42 +0100

[Robie Basak (racb)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Damiön la Bagh (kat-amsterdam)]

I'm going to upgrade a system from 14.04.5 to 16.04.x tomorrow at 10:00AM CEST so I'll let you know the results.

[Damiön la Bagh (kat-amsterdam)]

Ok so this is still FUBAR but then on a higher level.

The upgrade failed miserably see the attached file.
I was able to ssh into the machine after the upgrade.
ran dpkg --reconfigure -a as suggested see below

then I tried to reboot. which was a big mistake.
when I try to reconnect to the remote machine

ssh: connect to host remotemachine port 22: No route to host

This is terrible. ssh should be like the first thing started always. even grub should have some kind of ssh feature so if the grub menu shows one could fix that remotely.

Now I have a unusable machine and have to go physically to the location to try and reinstall everything by hand.

==============================
 sudo shutdown --reboot 1
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to start reboot.target: Failed to setup environment correctly
See system logs and 'systemctl status reboot.target' for details.
====================================================================

=====================================
It found too many errors to continue
Errors found when configuring:
 libvlc5
 libnss-winbind:amd64
 ppp
 network-manager-pptp
 cups
 cups-daemon
 cups-core-drivers
 libpam-winbind:amd64
 smbclient
 udev
 gnome-bluetooth
 libwayland-egl1-mesa:amd64
 samba
 vlc-plugin-notify
 samba-common-bin
 bluez
 indicator-bluetooth
 network-manager-pptp-gnome
 hplip
 pulseaudio
 mountall
 unity-control-center
 printer-driver-hpcups
 libcogl20:amd64
 initramfs-tools-core
 printer-driver-postscript-hp
 lvm2
 libtotem0:amd64
 libwebkit2gtk-4.0-37:amd64
 libcheese-gtk25:amd64
 libcanberra-pulse:amd64
 libsolid4
 initramfs-tools
 udisks2
 upower
 xserver-xorg-input-vmmouse
 gnome-control-center
 libgtk-3-0:amd64
 xserver-xorg-core
 gstreamer1.0-plugins-bad:amd64
 winbind
 xserver-xorg-video-radeon
 libpeas-1.0-0:amd64
 plymouth
 xserver-xorg
 libgstreamer-plugins-bad1.0-0:amd64
 bamfdaemon
 xfce4-power-manager
 gnome-system-monitor
 network-manager
 kdelibs5-plugins
Stopped configuring too many errors
===================================

[Damiön la Bagh (kat-amsterdam)]

One quick little rant.

Landscape is supposed to be Enterprise Grade software. I should be able to upgrade the releases of an entire farm of machines. This has been broken since the first time I tried this feature with Landscape.

[Damiön la Bagh (kat-amsterdam)]

So physically at the machine it was only possible to log into the command prompt. There is no graphical interface available.

To top it off

Network manager was stopped! Not very good for network-manager to be stopped right after an upgrade.

sudo service network-manager restart

started the network, where I could resume fixing the computer over ssh.

sudo apt install -f

is my first course of action.

[Łukasz Zemczak (sil2100)]

Hello Damiön, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Damiön, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu6

---------------
landscape-client (18.01-0ubuntu6) disco; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor.
  * Fixes for release-upgrade (LP: #1699179).
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/unicode-tags-script.patch: Permit environments
    containing unicode chars for script execution. (LP: #1765518)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

[Dimitri John Ledkov (xnox)]

Testing systemd 237-3ubuntu10.10, with masked and stopped systemd-logind, `sudo shutdown +1` command proceeds to shutdown in a VM.

[Timo Aaltonen (tjaalton)]

An upload of landscape-client to cosmic-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

[Timo Aaltonen (tjaalton)]

An upload of landscape-client to bionic-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

[Timo Aaltonen (tjaalton)]

An upload of landscape-client to xenial-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

[Timo Aaltonen (tjaalton)]

An upload of landscape-client to trusty-proposed has been rejected from the upload queue for the following reason: "Includes a change without an SRU bug attached to it".

[Timo Aaltonen (tjaalton)]

Hello Damiön, or anyone else affected,

Accepted landscape-client into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Damiön, or anyone else affected,

Accepted landscape-client into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Damiön, or anyone else affected,

Accepted landscape-client into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu2.16.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Damiön, or anyone else affected,

Accepted landscape-client into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/14.12-0ubuntu6.14.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Simon Poirier (simpoir)]

I tested successfully landscape-client and landscape-common packages from {trusty,xenial,bionic,cosmic}-proposed. As in the test case, masked systemd-logind.

After triggering reboots from landscape on all of those. All machines both force-rebooted as would be expected, and surfaced the error to the web interface
https://screenshots.firefox.com/bCPz5KxKkx4yG5hl/landscape.canonical.com

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu4.1

---------------
landscape-client (18.01-0ubuntu4.1) cosmic; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor. (LP: #1788219)
  * Fixes for release-upgrade:
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/unicode-tags-script.patch: Permit environments
    containing unicode chars for script execution. (LP: #1765518)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 18.01-0ubuntu3.2

---------------
landscape-client (18.01-0ubuntu3.2) bionic; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor. (LP: #1788219)
  * Fixes for release-upgrade:
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/unicode-tags-script.patch: Permit environments
    containing unicode chars for script execution. (LP: #1765518)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 16.03-0ubuntu2.16.04.5

---------------
landscape-client (16.03-0ubuntu2.16.04.5) xenial; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor (LP: #1788219)
  * Fixes for release-upgrade (LP: #1699179).
    - debian/patches/1699179-release-upgrade-check.diff: Check if ubuntu-
      release-upgrader is running before apt-update. (LP: #1699179)
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 14.12-0ubuntu6.14.04.4

---------------
landscape-client (14.12-0ubuntu6.14.04.4) trusty; urgency=medium

  * debian/patches/nutanix-kvm.patch: Update vm_info.py to include Nutanix
    hypervisor. (LP: #1788219)
  * Fixes for release-upgrade (LP: #1699179).
    - debian/patches/1699179-release-upgrade-check.diff: Check if ubuntu-
      release-upgrader is running before apt-update. (LP: #1699179)
    - debian/patches/release-upgrade-success.patch: Enable landscape-client to
      survive trusty upgrade. (LP: #1670291)
    - debian/patches/post-upgrade-reboot.patch: Force reboot operation in
      case systemd fails. (LP: #1670291)
  * debian/patches/1616116-resync-loop.patch:
    Clear hash id database on package resync. (LP: #1616116)

 -- Simon Poirier <email address hidden> Tue, 27 Nov 2018 09:24:22 -0500

[Steve Langasek (vorlon)]

Hello Damiön, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.30 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Damiön, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.31 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.31) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (ppc64el, amd64)
netplan.io/0.97-0ubuntu1~18.04.1 (amd64)
apt/1.6.12 (arm64, ppc64el)
pulseaudio/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Damiön la Bagh (kat-amsterdam)]

I saved one 14.04.5 machine for this purpose.
Will I be able to enabled -proposed for 14.04.5 even though it's now out of support?

[Dimitri John Ledkov (xnox)]

@kat-amsterdam

The fixes we are introducing are for upgrades to 18.04, which by default is not possible from 14.04.

Upgrade from 14.04 to 16.04 (trusty to xenial) has already been fixed a while back in landscape client, thus if one uses uptodate client, upgrades should succeed and reboot without any issues.

[Balint Reczey (rbalint)]

verified with 237-3ubuntu10.31:

root@bb-lp1670291:~# dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-======================-================-================-==================================================
ii systemd 237-3ubuntu10.31 amd64 system and service manager
root@bb-lp1670291:~# systemctl mask systemd-logind.service
Created symlink /etc/systemd/system/systemd-logind.service → /dev/null.
root@bb-lp1670291:~# systemctl stop systemd-logind.service
root@bb-lp1670291:~# shutdown +1
Failed to set wall message, ignoring: Unit systemd-logind.service is masked.
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Unit systemd-logind.service is masked.

Session terminated, terminating shell...rbalint@yogi:~$ lxc shell bb-lp1670291
Error: Container is not running
rbalint@yogi:~$ lxc shell bb-lp1670291
Error: Container is not running
rbalint@yogi:~$ lxc start bb-lp1670291
rbalint@yogi:~$ lxc shell bb-lp1670291
mesg: ttyname failed: No such device
root@bb-lp1670291:~# uptime
 12:23:38 up 0 min, 0 users, load average: 3.02, 2.14, 1.86
root@bb-lp1670291:~#

with unfixed reference version:
rbalint@yogi:~$ lxc shell bb-lp1670291-ref
mesg: ttyname failed: No such device
root@bb-lp1670291-ref:~# systemctl mask systemd-logind.service
Created symlink /etc/systemd/system/systemd-logind.service → /dev/null.
root@bb-lp1670291-ref:~# systemctl stop systemd-logind.service
root@bb-lp1670291-ref:~# shutdown +1
Failed to set wall message, ignoring: Unit systemd-logind.service is masked.
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Unit systemd-logind.service is masked.
root@bb-lp1670291-ref:~# dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-================-================-===================================================
ii systemd 237-3ubuntu10.29 amd64 system and service manager
root@bb-lp1670291-ref:~# uptime
 12:23:49 up 6 min, 0 users, load average: 2.70, 2.10, 1.85
root@bb-lp1670291-ref:~#

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.31

---------------
systemd (237-3ubuntu10.31) bionic; urgency=medium

  [ Dimitri John Ledkov ]
  * Add conflicts with upstart and systemd-shim. (LP: #1773859)
  * d/p/debian/UBUNTU-units-disable-journald-watchdog.patch
    - units: Disable journald Watchdog (LP: #1773148)
  * d/p/cryptsetup-add-support-for-sector-size-option-8881.patch
    - cryptsetup: add support for sector-size= option (LP: #1776626)
  * d/p/systemctl-correctly-proceed-to-immediate-shutdown-if-sche.patch
    - systemctl: correctly proceed to immediate shutdown if scheduling fails
      (LP: #1670291)
  * d/p/networkd-add-support-to-configure-IPv6-MTU-8664.patch
    - networkd: add support to set IPv6MTUBytes (LP: #1671951)

 -- Balint Reczey <email address hidden> Mon, 30 Sep 2019 17:23:17 +0200