release upgrade from xenial to bionic desktop: screen locks itself, password to unlock fails

[Steve Langasek]

[Impact]
During a release upgrade, the screen should not be locked because the upgrade of underlying libraries may leave the system in an inconsistent state where the process locking the screen may not be able to unlock it again. (Its possible for the screen to lock during the fetching process before the call to inhibitIdle happens.)

[Test Case]
1) Configure a screen lock with a short time period (1 minute) and ensure it'll take more than 1 minute to fetch all the packages
2) Upgrade an Ubuntu 16.04 system to Ubuntu 18.04
3) Observe the screen locks during the upgrade process

With the version of ubuntu-release-upgrader from -proposed the screen will not lock during the fetching process. I wasn't able to recreate the bug where the system would not unlock but testing that the screen doesn't lock should be sufficient to verify this.

[Regression Potential]
Code is just being moved from DistUpgradeQuirks.py to DistUpgradeController.py but its possible a mistake was made so this should be tested carefully.

[Original Description]
There is code in the ubuntu-release-upgrader to handle this (DistUpgrade/DistUpgradeQuirks.py:DistUpgradeQuirks._inhibitIdle).

I have just started a release upgrade of an Ubuntu desktop from xenial to bionic with update-manager -d. After leaving it unattended for a while, I came back to find the screen was locked.

/var/log/auth.log includes messages such as:

Jun 26 16:00:45 epona compiz: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory

This indicates a problem dlopen()ing the PAM modules (actual path: /lib/x86_64-linux-gnu/security/pam_unix.so) because some ABI has changed in the system libraries, and the copy loaded into the running compiz process does not have the symbols required in order to run this module.

This is the exact reason the screensaver is supposed to be inhibited on upgrade.

This is also why every sensible screensaver spawns a fresh helper process to handle the authentication through PAM. compiz (unity), apparently, does not.

I do not yet know what symbols have changed to cause this failure. I'm still investigating that.

[Steve Langasek (vorlon)]

And then the upgrade gets stuck because of a conffile prompt.

[Steve Langasek (vorlon)]

The failure is because pam_unix depends on libcrypt, and libcrypt is part of glibc, and compiz doesn't otherwise have libcrypt loaded into memory, and libcrypt has a tight versioned dependency on the corresponding version of libc itself (requires symbol GLIBC_2.25). So when dlopen() tries to load libcrypt from disk, because compiz is doing it wrong, it can't find the symbols.

[Julian Andres Klode (juliank)]

So, u-r-u runs /usr/bin/gnome-session-inhibit --inhibit idle --inhibit-only - running that manually works fine to block stuff on cosmic, but not sure about xenial.

systemd-inhibit --what=idle does not prevent locking.

[Sebastien Bacher (seb128)]

That's the first report of the issue and screen locking inhibition has been working fine on xenial so far, is there anything useful (warning, errors, info about screen locking, inhibiting, screensaver, session) in the system logs/journal? Can you reproduce the bug or was it a one time thing?

[Sebastien Bacher (seb128)]

Doing "/usr/bin/gnome-session-inhibit --inhibit idle --inhibit-only" on Xenial seems to work fine so the issue seems to be on the ubuntu-release-upgrader side. Is the process using the correct user environment (uid, dbus session bus, etc) to talk to the active session?

[Sebastien Bacher (seb128)]

@Steve, you are probably right about the compiz use of pam but that doesn't seem likely to get changed at this point, at least not before the LTS upgrades to bionic are turned on

[Steve Langasek (vorlon)]

On Tue, Jul 03, 2018 at 02:26:14PM -0000, Sebastien Bacher wrote:
> @Steve, you are probably right about the compiz use of pam but that
> doesn't seem likely to get changed at this point, at least not before
> the LTS upgrades to bionic are turned on

Ok, but we still need to know how to inhibit the screen locking in compiz,
since the previous standard method is ineffective.

[Brian Murray (brian-murray)]

I believe the issue is that the gnome-session-inhibit call is terminated after the upgrade unless you are running a GNOME session.

https://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-release-upgrader/trunk/view/head:/DistUpgrade/DistUpgradeQuirks.py#L444

After the upgrade completed and choosing not to reboot I was presented with a lock screen that continually said "Authentication failed".

Did your upgrade complete and then the screen locked?

[Steve Langasek (vorlon)]

No. This upgrade had the screen locked over a conffile prompt. The upgrade was definitely still in progress.

[Jean-Baptiste Lallement (jibel)]

I reproduced the second part of the problem (screen locks and fails to unlock) after upgrading from 16.04 to 18.04 and before restarting the machine (finish the upgrade, close the upgrader and wait until the screen saver goes off)

The following lines are displayed in the journal:
juil. 04 09:27:07 ubuntu compiz[1692]: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
juil. 04 09:27:07 ubuntu compiz[1692]: PAM adding faulty module: pam_unix.so
juil. 04 09:27:07 ubuntu compiz[1692]: PAM unable to dlopen(pam_systemd.so): /lib/security/pam_systemd.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
juil. 04 09:27:07 ubuntu compiz[1692]: PAM adding faulty module: pam_systemd.so

However I didn't reproduce the "lock screen during upgrade" issue even with a config prompt during upgrade.

[Sebastien Bacher (seb128)]

> Ok, but we still need to know how to inhibit the screen locking in compiz,
> since the previous standard method is ineffective.

The command that Julian pointed out as the one used by u-r-u is correct and works from our testing

> unless you are running a GNOME session.

> https://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-release-upgrader/trunk/view/head:/DistUpgrade/DistUpgradeQuirks.py#L444

It looks like Unity sessions are in a similar situation so maybe the code should be change to keep the inhibitor for those as well (maybe poping up a dialog explaining that screen lock is not going to work until reboot if that's less confusing to users?). That's a different case than the one reported here though

[Brian Murray (brian-murray)]

On Wed, Jul 04, 2018 at 07:47:15AM -0000, Sebastien Bacher wrote:
> > Ok, but we still need to know how to inhibit the screen locking in compiz,
> > since the previous standard method is ineffective.
>
> The command that Julian pointed out as the one used by u-r-u is correct
> and works from our testing
>
>
> > unless you are running a GNOME session.
>
> > https://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-release-
> upgrader/trunk/view/head:/DistUpgrade/DistUpgradeQuirks.py#L444
>
> It looks like Unity sessions are in a similar situation so maybe the
> code should be change to keep the inhibitor for those as well (maybe
> poping up a dialog explaining that screen lock is not going to work
> until reboot if that's less confusing to users?). That's a different
> case than the one reported here though

Right, I'll open a new bug (or look for an existing one) regarding the
Unity session locking after upgrade issue on Thursday.

--
Brian Murray

[Brian Murray (brian-murray)]

Steve - your /var/log/dist-upgrade/main.log file may be helpful in sorting out where the inhibit went wrong. Could you add it to the bug?

[Jean-Baptiste Lallement (jibel)]

I can reproduce first part of the problem during an upgrade from 17.10 to 18.04 and the screen locks during the upgrade. It locks before u-r-u issues the inhibit command and proceeds with the upgrade with the screen locked.

[Steve Langasek (vorlon)]

syslog search for 'inhibit':

$ zgrep -i inhibit /var/log/syslog.*
/var/log/syslog.5.gz:Jun 26 11:56:40 epona org.freedesktop.Notifications[1961]: ** (notify-osd:2212): WARNING **: dnd_is_idle_inhibited(): got error "No such method 'IsInhibited'"
$

so, not interesting.

/var/log/dist-upgrade/main.log attached. Critical section is:

2018-06-26 15:30:27,637 DEBUG found components: {'bionic': {'universe', 'multiverse', 'main', 'restricted'}, 'bionic-security': {'universe', 'multiverse', 'main', 'restricted'}, 'bionic-updates': {'universe', 'multiverse', 'main', 'restricted'}}
2018-06-26 15:55:19,566 DEBUG plugins for condition 'StartUpgrade' are '[]'
2018-06-26 15:55:19,567 DEBUG plugins for condition 'bionicStartUpgrade' are '[]'
2018-06-26 15:55:19,567 DEBUG plugins for condition 'from_xenialStartUpgrade' are '[]'
2018-06-26 15:55:19,567 DEBUG quirks: running StartUpgrade
[...]
2018-06-26 15:55:19,571 DEBUG killing update-notifier
2018-06-26 15:55:19,587 DEBUG killing kblueplugd kbluetooth4
2018-06-26 15:55:19,605 DEBUG killing gnome-screensaver
2018-06-26 15:55:19,614 DEBUG setup poke timer for the scrensaver
2018-06-26 15:55:19,626 DEBUG inhibit gnome-session idle
2018-06-26 15:55:19,630 DEBUG apt btrfs snapshots supported: False
2018-06-26 15:55:19,630 INFO cache.commit()
2018-06-26 15:55:19,630 DEBUG failed to SystemUnLock() (E:Not locked)
2018-06-26 16:17:28,618 DEBUG got a conffile-prompt from dpkg for file: '/etc/ssh/ssh_config'

I'm not sure, could the problem be that the inhibit was not emitted until long after the last prompt before the upgrade started?

[Brian Murray (brian-murray)]

I reported bug 1780342 regarding the lock screen after the upgrade has completed and one has not rebooted.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.4

---------------
ubuntu-release-upgrader (1:18.10.4) cosmic; urgency=medium

  * DistUpgradeQuirks.py, DistUpgradeController.py: move inhibiting of
    gnome-session idle so that it is called immediately after the upgrade
    starts (LP: #1778817), present a dialog notifying that the lock screen has
    been disabled (LP: #1174093), and do not reenable the lock screen as it
    may fail to work after a release upgrade. (LP: #1780342)
  * DistUpgradeQuirks.py: Add in a quirk for upgrading to Cosmic that will
    check if the system has connectivity to the snap store and replace some
    desktop packages with snaps to ensure the system is similar to a fresh
    install.

 -- Brian Murray <email address hidden> Fri, 06 Jul 2018 11:21:11 -0700

[Łukasz Zemczak (sil2100)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I tested a release upgrade from Ubuntu 16.04 to Ubuntu 18.04 today using the version of release-upgrader in -proposed and the lock screen did not appear.

bdmurray@clean-xenial-amd64:/tmp/u-r-u$ grep "release-upgrader version" /var/log/dist-upgrade/main.log
2018-07-16 14:00:39,174 INFO release-upgrader version '18.04.21' started

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.21

---------------
ubuntu-release-upgrader (1:18.04.21) bionic; urgency=medium

  * DistUpgradeQuirks.py: when a snap fails to install try the next one
    instead of trying to check the return code of the called process which
    causes a Traceback. (LP: #1781412)
  * data/mirrors.cfg: Update the list of official mirrors.

ubuntu-release-upgrader (1:18.04.20) bionic; urgency=medium

  * Purge obsolete packages, if user agrees to remove obsolete
    packages. LP: #1775660
  * DistUpgradeQuirks.py, DistUpgradeController.py: move inhibiting of
    gnome-session idle so that it is called immediately after the upgrade
    starts (LP: #1778817), present a dialog notifying that the lock screen has
    been disabled (LP: #1174093), and do not reenable the lock screen as it
    may fail to work after a release upgrade. (LP: #1780342)
  * DistUpgradeQuirks.py: Add in a quirk for upgrading to Bionic that will
    check if the system has connectivity to the snap store and replace some
    desktop packages with snaps to ensure the system is similar to a fresh
    install. (LP: #1780841)
  * data/mirrors.cfg: Update the list of official mirrors.

 -- Brian Murray <email address hidden> Sun, 15 Jul 2018 10:16:16 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andrea Azzarone (azzar1)]

Marking compiz as invalid because this is a ubuntu-release-upgrader problem.

[Steve Langasek (vorlon)]

It is not invalid; it is a regression vs all previous screen lock implementations that compiz does pam handling in process instead of in a helper process, making it vulnerable to this kind of library skew.