Upgrade from Ubuntu 21.10 to 22.04 fails - unmet dependencies: libpam-modules

[bugproxy]

[Impact]
Upgrade from Ubuntu 21.10 to 22.4 fails if pam_tally is present in the system's PAM config

I run do-release-upgrade and I got a message upgrade completed with errors. I rebooted the server and it is now in an undefined state between 21.10 and 22.04. Not all packages have been installed. I attach apt log and output of some commands:

root@tuxmaker:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 21.10"
NAME="Ubuntu"
VERSION_ID="21.10"
VERSION="21.10 (Impish Indri)"
VERSION_CODENAME=impish
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=impish

root@tuxmaker:~# do-release-upgrade
Checking for a new Ubuntu release
Please install all available updates for your release before upgrading.

root@tuxmaker:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libpam-modules : PreDepends: libpam-modules-bin (= 1.3.1-5ubuntu11) but 1.4.0-11ubuntu2 is installed
                  Recommends: update-motd but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

root@tuxmaker:~# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libpam-modules
The following packages will be upgraded:
  libpam-modules
1 upgraded, 0 newly installed, 0 to remove and 1208 not upgraded.
4 not fully installed or removed.
Need to get 0 B/279 kB of archives.
After this operation, 1,024 B disk space will be freed.
Do you want to continue? [Y/n] y
Preconfiguring packages ...
(Reading database ... 401827 files and directories currently installed.)
Preparing to unpack .../libpam-modules_1.4.0-11ubuntu2_s390x.deb ...
dpkg: error processing archive /var/cache/apt/archives/libpam-modules_1.4.0-11ubuntu2_s390x.deb (--unpack):
 new libpam-modules:s390x package pre-installation script subprocess returned error exit status 2
Errors were encountered while processing:
 /var/cache/apt/archives/libpam-modules_1.4.0-11ubuntu2_s390x.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Contact Information = <email address hidden>

[Test plan]
Test 1
------
The pre-build.sh script run prior to generation of the upload tarball runs the test suite which includes an added test for this quirk to verify its correct operation.

Test 2
------
Using the dist-upgrader tarball from http://archive.ubuntu.com/ubuntu/dists/jammy-proposed/main/dist-upgrader-all/current/ on an Ubuntu 20.04 LTS system upgrade to Ubuntu 22.04 LTS. (Using the tarball is necessary because one cannot use 'do-release-upgrade -d and -p' simultaneously.) The tarball should be manually extracted into /tmp/ and the file (or symlink) named 'jammy' should be executed.

Verify that the upgrade completes successfully with the addition of the new quirk, which should have no effect on a default install.

Test 3
------
On an impish system, run 'do-release-upgrade -p' to use the tarball from -proposed. Verify that the upgrade completes successfully.

[Where problems could occur]
The addition of a quirk could prevent some users from upgrading, in which case a further SRU will be necessary. The new code is run very early in the upgrader and it should not be possible for the new code to leave the system in an inconsistent state mid-upgrade.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Frank Heimes (fheimes)]

First of all I assume there was no previous attempt of doing a 'do-release-upgrade' or even an attempt that was stopped/interrupted, or an attempt to get the system updated in a different way, for example (in the abs. not recommended way to) to change all 'impish' entries in /etc/apt/sources.list to jammy or so.

Launchpad (https://launchpad.net/ubuntu/+source/pam/) clearly shows that
21.10/impish has currently 1.3.1-5ubuntu11 (and 22.04/jammy has 1.4.0-11ubuntu2).
A different way to check this is:
$ rmadison --arch=s390x libpam-modules | egrep "impish|jammy"
 libpam-modules | 1.3.1-5ubuntu11 | impish | s390x
 libpam-modules | 1.4.0-11ubuntu2 | jammy | s390x
(And since no 'impish-updates' line is listed here, this particular package was also never updated during the impish release.)

So since your current system is (still) 21.10, it must have version 1.3.1-5ubuntu11 installed.
$ lsb_release -cs
impish
$ apt-cache policy libpam-modules
libpam-modules:
  Installed: 1.3.1-5ubuntu11
  Candidate: 1.3.1-5ubuntu11
  Version table:
 *** 1.3.1-5ubuntu11 500
        500 http://ports.ubuntu.com/ubuntu-ports impish/main s390x Packages
        100 /var/lib/dpkg/status
... but I must not have 1.4.0-11ubuntu2 installed, since this is the version of the 22.04 release.

Calling 'apt-cache policy libpam-modules' on your system will give a hint where the currently installed version was taken from.
Sharing the output of this command from your system would be interesting - in addition to the 'sudo apt update' and 'lsb_release -a' output and the file /etc/apt/sources.list.

Such a situation could have been caused if someone downloaded the jammy version of a package manually and (forcefully) installed it
or if someone wanted to try packages from a different release and added an archive line pointing to that different Ubuntu release (e.g. jammy) to the /etc/apt/sources.list file.

The line "'1 upgraded, 0 newly installed, 0 to remove and 1208 not upgraded." is also very suspicious, because 1208 must be close to the overall number of packages in your system (a default install on s390x has about 1020 packages).

So please check that no packages are taken from an Ubuntu release other than the currently installed one:
$ lsb_release -cs
impish
Means check the content of /etc/apt/sources.list (and maybe further sources* files in that folder).
You should also monitor the output of 'sudo apt update' while updating the package indices and check if a wrong Ubuntu release is listed.

In case a wrong archive line is listed, fix it by editing the file and execute:
$ sudo apt clean && sudo apt update
If everything is fine now, upgrade the current system to the very latest level:
$ sudo apt full-upgrade # or maybe: apt dist-upgrade
before re-doing the 'do-release-upgrade'.

(IF a previous do-release-upgrade broke, or an attempt was made to upgrade the system in a different way, the system might be in a pretty bad shape and difficult to fix.
And 'sudo apt-get install --fix-broken' might be needed or even a forceful remove of a wrong package.
Hence, as long as you have access to this system, backup any important data!)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-06-03 08:33 EDT-------
I have already run the command do-release-upgrade. At the end of the upgrade I could see "upgrade completed" but there were errors. Than I rebooted the system.

I attach the sources.list file an the output of the commands.

root@tuxmaker:~# apt-cache policy libpam-modules
libpam-modules:
Installed: 1.3.1-5ubuntu11
Candidate: 1.4.0-11ubuntu2
Version table:
1.4.0-11ubuntu2 500
500 http://ports.ubuntu.com:80 jammy/main s390x Packages
*** 1.3.1-5ubuntu11 100
100 /var/lib/dpkg/status

root@tuxmaker:~# sudo apt update
Hit:1 http://ports.ubuntu.com:80 jammy InRelease
Get:2 http://ports.ubuntu.com:80 jammy-updates InRelease [109 kB]
Get:3 http://ports.ubuntu.com:80 jammy-backports InRelease [99.8 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [110 kB]
Get:5 http://ports.ubuntu.com:80 jammy-updates/main s390x Packages [170 kB]
Get:6 http://ports.ubuntu.com:80 jammy-updates/main Translation-en [59.5 kB]
Get:7 http://ports.ubuntu.com:80 jammy-updates s390x Contents (deb) [3,398 kB]
Get:8 http://ports.ubuntu.com:80 jammy-updates/main s390x c-n-f Metadata [3,904 B]
Get:9 http://ports.ubuntu.com:80 jammy-updates/restricted s390x Packages [3,024 B]
Get:10 http://ports.ubuntu.com:80 jammy-updates/restricted Translation-en [21.4 kB]
Get:11 http://ports.ubuntu.com:80 jammy-updates/universe s390x Packages [68.4 kB]
Get:12 http://ports.ubuntu.com:80 jammy-backports s390x Contents (deb) [12.6 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-security/main s390x Packages [77.2 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports jammy-security/main Translation-en [34.0 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-security s390x Contents (deb) [2,164 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports jammy-security/main s390x c-n-f Metadata [2,132 B]
Fetched 6,334 kB in 1s (5,198 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1209 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@tuxmaker:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.10
Release: 21.10
Codename: impish

There are over 2000 Packages on the system:
root@tuxmaker:~# dpkg-query -l | wc -l
2061

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2022-06-03 08:34 EDT-------

[Frank Heimes (fheimes)]

Well, it's a bit difficult to give remote recommendation for a system in this state.
(Ideally I want to login to it by myself ...)

It would be good to know how many of the installed packages are still packages from impish, and how many already from jammy.

One, may try to move forward to jammy (and forcing it),
but I think it's better to be careful and go one step back first of all.

So I would replace all occurrences of jammy in /etc/apt/sources.list back to impish
and do an 'apt clean' and 'apt update'.

Now I provoked a /similar/ situation like yours by manually downloading the jammy package and force installing it:

$ wget http://launchpadlibrarian.net/592653998/libpam-modules-bin_1.4.0-11ubuntu2_s390x.deb

$ sudo dpkg -i --force-depends ./libpam-modules-bin_1.4.0-11ubuntu2_s390x.deb
(Reading database ... 83842 files and directories currently installed.)
Preparing to unpack .../libpam-modules-bin_1.4.0-11ubuntu2_s390x.deb ...
Unpacking libpam-modules-bin (1.4.0-11ubuntu2) over (1.3.1-5ubuntu11) ...
Setting up libpam-modules-bin (1.4.0-11ubuntu2) ...
Processing triggers for man-db (2.9.4-2) ...

After that apt full-upgrade shows the same error like on your system:

$ sudo apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libpam-modules : PreDepends: libpam-modules-bin (= 1.3.1-5ubuntu11) but 1.4.0-11ubuntu2 is installed
                  Recommends: update-motd but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Now - since we want to go back to impish - the opposite can be done to solve the situation, means forcing the installation of the impish package:

$ wget http://launchpadlibrarian.net/558628390/libpam-modules-bin_1.3.1-5ubuntu11_s390x.deb

$ sudo dpkg -i --force-depends ./libpam-modules-bin_1.3.1-5ubuntu11_s390x.deb
dpkg: warning: downgrading libpam-modules-bin from 1.4.0-11ubuntu2 to 1.3.1-5ubuntu11
(Reading database ... 83840 files and directories currently installed.)
Preparing to unpack .../libpam-modules-bin_1.3.1-5ubuntu11_s390x.deb ...
Unpacking libpam-modules-bin (1.3.1-5ubuntu11) over (1.4.0-11ubuntu2) ...
Setting up libpam-modules-bin (1.3.1-5ubuntu11) ...
Processing triggers for man-db (2.9.4-2) ...

After that apt is happy again:

$ sudo apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  apport-symptoms eatmydata libavahi-core7 libdaemon0 libeatmydata1
  libregexp-assemble-perl libwrap0 python3-debconf python3-jinja2
  python3-json-pointer python3-jsonpatch python3-jsonschema python3-markupsafe
  python3-pyrsistent python3-systemd squashfs-tools
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

(Please notice that forcing packages is usually a bad idea and can break the packaging system, and should only be done with caution and in rare corner cases!...

[Steve Langasek (vorlon)]

If there's any bug in Ubuntu here, it's that the release upgrader didn't try harder to get the system into a consistent state before offering a reboot. No bug in pam, which is simply one of the packages to be upgraded.

libpam-modules refused to be upgraded because there were references to obsolete modules in the system config and one or more services would be broken after upgrade. At the per package level there really isn't a better way to handle this.

[Frank Heimes (fheimes)]

Fwiw - I just took the system that I used for the description above and did a 'do-release-upgrade' and it work fine - in my admittedly relatively clean system - and the upgrade completed successfully (https://pastebin.ubuntu.com/p/KcgVj3y5xT/).
(But of course not all systems are the same - and I say that your's was once a bionic system - hence already upgraded a couple of times...)

[Steve Langasek (vorlon)]

On Fri, Jun 03, 2022 at 03:58:24PM -0000, Frank Heimes wrote:

> Fwiw - I just took the system that I used for the description above and
> did a 'do-release-upgrade' and it work fine

Yes, this problem only occurs because the submitter has references to the
obsolete pam_tally module in their config and PAM blocks the upgrade to
prevent the user from having a broken config after upgrade (and potentially
be locked out of administering the system to fix it). It will not happen on
a system with a stock config.

It is unfortunately not safe for us to automatically edit the configs to
remove references to pam_tally on upgrade, as this may also result in an
admin lock-out.

[Steve Langasek (vorlon)]

I'm going to mark this triaged for ubuntu-release-upgrader. It's important to have guards in preinst scripts against breaking a system as part of an upgrade, but we also want to avoid actually hitting these failures in the middle of a release upgrade precisely because it's so difficult to get the system back into a coherent state afterwards. Where we know such preinst checks exist (especially for such critical packages as pam), it's best to have do-release-upgrader detect early and block the upgrade.

[Frank Heimes (fheimes)]

To sum this up and to unblock:

The root cause of your issue is the use of pam_tally (and the non-optimal way this situation is currently handled by do-release-upgrade), like mentioned by vorlon (thx) in the previous comment(s) and also indicated in the log you've provided.

Since either the pam_tally or the pam_tally2 module is in use by your system (in /etc/pam.d),
but pam_tally became obsolete and got meanwhile removed from PAM,
you must remove pam_tally before PAM can be properly upgraded.
(And notice that including these modules in your PAM configuration again after the upgrade will stop users from being able to log into the system.)
Consider pam_faillock as potential replacement for pam_tally.)

This needs to be manually done before running do-release-upgrade again.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-06-07 09:47 EDT-------
Thanks for the hints, we will continue to follow them up after the Pentecost holidays when all employees are back at work, as long as the server stays in the current state. I have set the priority of the ticket to "P3" "normal".

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-06-07 11:45 EDT-------
Thanks for the hints, we will continue to follow them up after the Pentecost holidays when all employees are back at work, as long as the server stays in the current state. I have set the priority of the ticket to "P3" "normal".

------- Comment From <email address hidden> 2022-06-07 11:46 EDT-------
Thanks for the hints, we will continue to follow them up after the Pentecost holidays when all employees are back at work, as long as the server stays in the current state. I have set the priority of the ticket to "P3" "normal".

------- Comment From <email address hidden> 2022-06-07 11:47 EDT-------
Thanks for the hints, we will continue to follow them up after the Pentecost holidays when all employees are back at work, as long as the server stays in the current state. I have set the priority of the ticket to "P3" "normal".

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2022-07-01 09:09 EDT-------
Finally I managed to get a maintenance window for the server and try your suggestions. I removed the pam_tally2 from the pam configuration, downgraded the libpam-modules-bin package and changed the sources.list to impish. After that I was able to successfully upgrade the server to Ubuntu 22.04. Thank you very much for the suppor!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Halse Rogers (raof)]

An upload of ubuntu-release-upgrader to jammy-proposed has been rejected from the upload queue for the following reason: "Contains Python cache detritus".

[Chris Halse Rogers (raof)]

Hello bugproxy, or anyone else affected,

Accepted ubuntu-release-upgrader into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:22.04.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

SRU verification:

$ lxc launch ubuntu:focal lp1977493
Creating lp1977493
Starting lp1977493
$ lxc exec lp1977493 bash
root@lp1977493:~# echo -e "auth\trequired\tpam_tally.so" >> /etc/pam.d/login
root@lp1977493:~# cd /tmp/
root@lp1977493:/tmp# wget -q http://archive.ubuntu.com/ubuntu/dists/jammy-proposed/main/dist-upgrader-all/current/jammy.tar.gz
root@lp1977493:/tmp# tar xf jammy.tar.gz
root@lp1977493:/tmp# ./jammy
[...]
Reading package lists... Done
Building dependency tree
Reading state information... Done

You are using pam_tally or pam_tally2 in your configuration.

The pam_tally and pam_tally2 modules have been removed from PAM. You
are using one of these modules in your PAM configuration in
/etc/pam.d. You must remove the uses of these modules before PAM can
be upgraded; including these modules in your PAM configuration after
the upgrade will stop users from being able to log into the system.

Consider the pam_faillock module as a replacement for pam_tally.

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command terminated with exit status 1 (Wed Aug 3 04:43:30 2022) ===
[screen is terminating]
root@lp1977493:/tmp# sed -i -e'/pam_tally/d' /etc/pam.d/login
root@lp1977493:/tmp# ./jammy
[...]

Reading package lists... Done
Building dependency tree
Reading state information... Done

Calculating the changes

Calculating the changes

Do you want to start the upgrade?

4 packages are going to be removed. 85 new packages are going to be
installed. 538 packages are going to be upgraded.

You have to download a total of 188 M. This download will take about
37 seconds with a 40Mbit connection and about 5 minutes with a 5Mbit
connection.

Fetching and installing the upgrade can take several hours. Once the
download has finished, the process cannot be canceled.

 Continue [yN] Details [d]y
[...]
Searching for obsolete software
Reading state information... Done

Remove obsolete packages?

72 packages are going to be removed.

 Continue [yN] Details [d]y
[...]
System upgrade is complete.

Restart required

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN]
[screen is terminating]
root@lp1977493:/tmp# halt -p
$ lxc delete lp1977493
$ lxc launch ubuntu:impish lp1977493
Creating lp1977493
Starting lp1977493
$ lxc exec lp1977493 bash
root@lp1977493:~# echo -e "auth\trequired\tpam_tally.so" >> /etc/pam.d/login
root@lp1977493:~# do-release-upgrade -p
Checking for a new Ubuntu release
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
[...]
Continue [yN] y
[...]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

You are using pam_tally or pam_tally2 in your configuration.

The pam_tally and pam_tally2 modules have been removed from PAM. You
are using one of these modules in your PAM configuration in
/etc/pam.d. You must remove the uses of these modules before PAM can
be upgraded; including these modules in your PAM co...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:22.04.13

---------------
ubuntu-release-upgrader (1:22.04.13) jammy; urgency=medium

  [ Simon Chopin ]
  * Add a quirk to resolve an issue where the nvidia driver would get
    suggested for autoremoval. (LP: #1955047)

  [ Steve Langasek ]
  * On upgrade to 22.04, detect the presence of pam_tally* in /etc/pam.d and
    block the upgrade early, to avoid an abort from libpam-modules in the
    middle of the apt upgrade. LP: #1977493.
  * Run pre-build.sh: updating mirrors and translations.

 -- Steve Langasek <email address hidden> Tue, 02 Aug 2022 17:15:07 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Abhijith NT (abhint7)]

└─$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libpam-modules
The following packages will be upgraded:
  libpam-modules
1 upgraded, 0 newly installed, 0 to remove and 42 not upgraded.
Need to get 0 B/280 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable
dpkg: warning: files list file for package 'libpaper1:amd64' missing; assuming package has no files currently inst
alled
(Reading database ... 272799 files and directories currently installed.)
Preparing to unpack .../libpam-modules_1.4.0-11ubuntu2.1_amd64.deb ...
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unava
ilable
dpkg: error processing archive /var/cache/apt/archives/libpam-modules_1.4.0-11ubuntu2.1_amd64.deb (--unpack):
 new libpam-modules:amd64 package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/libpam-modules_1.4.0-11ubuntu2.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)