Quirks: Add upgrade quirk for FIPS enabled systems to work around libgcrypt hmac file placement

[Matthew Ruffell]

[Impact]

When upgrading a Bionic fips (NOT fips-updates) system to Focal fips, the upgrade gets stuck generating the initramfs with the following message:

update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic
Failed to copy HMAC file "/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac".
E: /usr/share/initramfs-tools/hooks/fips-libgcrypt failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-84-generic with 1.
dpkg: error processing package ubuntu-fips (--configure):
 installed ubuntu-fips package post-installation script subprocess returned error exit status 1

This happens because we are trying to copy /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the initramfs, but because of a bug, it is actually installed to /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac.

Most normal Focal systems are not affected, due to being usrmerged [1], where /lib is a symlink to /usr/lib.

[1] https://wiki.debian.org/UsrMerge

But when we upgrade from Bionic, usrmerge is not performed.

We have resolved this bug in two new package releases for Focal, in fips-updates:
- ubuntu-fips (1.2.4+updates1)
- libgcrypt20 (1.8.5-5ubuntu1.fips.1.5)

This was a part of bug 1944403, which is private due to having debdiffs for FIPS packages attached to the bug. Ask mruffell for access if you wish to view the bug.

We changed /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be placed in /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac, and we changed the initramfs hook script to look for existence in both /usr/lib and /lib.

The problem is, libgcrypt20 (1.8.5-5ubuntu1.fips.1.5) did not make the current re-certification round and Canonical have already received a certificate for libgcrypt20 (1.8.5-5ubuntu1.fips.1.4) stating fips compliance, thus we cannot get 1.8.5-5ubuntu1.fips.1.5 certified in this current round.

The next certification round may take place in a year or two, but we require a solution for users who wish to upgrade from Bionic fips to Focal fips, and those users cannot enable fips-updates for strict compliance reasons.

The workaround is to add a dpkg-divert, somewhat of the form of:

$ [ ! -L /lib ] && sudo dpkg-divert --local --rename --add --divert /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac

I have implemented this as a upgrade quirk in DistUpgradeQuirks.py, in the function _fipsLibgcryptDivert(), which checks for the presence of libgcrypt20-hmac and checking to see if the broken fips certified package would be installed, before adding the dpkg-divert.

Note, I did not use the --rename option in the quirk, as we require /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be left in place while we are performing the upgrade, as the libgcrypt self-test will fail if the file is removed before we install the Focal libgcrypt package. The error is very noisy as it is printed each time a program is invoked that uses libgcrypt, e.g. apt, dpkg, and it could prevent the upgrade from completing successfully.

libgcrypt selftest: binary (0): No such file or directory (/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)

Instead, we leave /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac in place until the very end of the upgrade, where we remove it in the function _fipsLibgcryptHmacCleanup().

[Testcase]

There are three upgrades that need to be tested to ensure things work as intended:

- Bionic
- Bionic fips
- Bionic fips-updates

Systems need to be attached to a UA token, and fips or fips-updates enabled and installed, before upgrade starts.

You can use do-release-upgrade to perform the upgrade.

The patch can be found here:

https://paste.ubuntu.com/p/FgrnsRfZPt/

You can test the upgrade before the merge to ubuntu-release-upgrader by:

$ sudo -s
# vim ~/quirk.patch
Stick the contents of the pastebin in the file.

# apt update
# apt upgrade
# do-release-upgrade

At the first screen where you are asked to start another ssh server, hit ctrl-c
and then press x to exit screen.

We need to patch the quirk file we just downloaded.

# cd /tmp
# cd ubuntu-release-upgrader-<RANDOM LETTERS>
# patch -p1 < ~/quirk.patch

Hopefully it applies cleanly.

We can restart do-release-upgrade with our patched copy with:

# ./focal

From there, all upgrades should succeed cleanly.

Ideally we would also test upgrading to Jammy from the resulting Focal install, to ensure usrmerge works as intended, and that /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac was cleaned up correctly.

[Where problems could occur]

We are adding a quirk to Bionic systems upgrading to Focal only. This quirk is not intended to be used on any platform other than Bionic->Focal, and thus only targets the Focal branch of ubuntu-release-upgrader.

We check to make sure that libgcrypt20-hmac is installed, and if the package is not installed, we exit from the quirk. This way we ensure we only apply the quirk to FIPS enabled systems.

We also exit from the quirk if fips-updates is being used, and the fixed libgcrypt20-hmac package is to be installed as part of the upgrade.

On cleanup, we take great care to ensure that we only remove the Bionic /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file, and leave the Focal /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file well alone. We ensure the system is not usrmerged, and check to ensure the files are not symlinks before removing the Bionic hmac file.

If a regression were to occur, it could prevent systems from upgrading correctly, and would have a significant impact to users before it would be fixed.

I believe I have narrowed the risk down to the small pool of fips users that require this quirk to be able to upgrade successfully.

[Robie Basak (racb)]

Some comments and questions on SRU review:

This is a bit special since it's not the usual "backport a fix" type SRU, so I thought that reviewing the code changes themselves seemed more important than usual in this case. So here are my code review comments:

Why are you calling subprocess.Popen() instead of subprocess.check_call() or similar? To use Popen() directly, you also need to call wait() on the output of subprocess.Popen() really and then check the result code. It'll work if you don't, but any problems will fail silently and won't be logged, so as written this seems buggy to me.

On check_call() vs. call(), I suggest check_call() since you're already catching, logging and then ignoring errors, and a failure would then be logged appropriately.

Presumably this script needs to work with the Python version shipped in Bionic? I checked and that seems to be 3.6, and https://docs.python.org/3/library/subprocess.html#subprocess.check_call says check_call() was introduced prior to 3.3, so I think you're good there.

Why are you invoking the rm command instead of calling os.unlink()?

Note that there are cases where two files are the same but not a symlink. A hard link is one, and that would be fine in this case. Another case that would not be fine is a bind mount. You might conclude that this case isn't reasonable to support; I'll leave that decision to you.

However, if you aren't checking hashes following review changes, then please update the bug description so that you're no longer saying that you are :)

[Matthew Ruffell (mruffell)]

Hi Robie,

Thank you for your feedback, it is all very reasonable, and I have actioned upon it, by changing the subprocess.Popen() call to check_call(), and changed the rm to os.unlink().

I opened a new merge request at https://code.launchpad.net/~mruffell/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/435410 with the changes.

[Brian Murray (brian-murray)]

I've merged Matthew's changes and uploaded a new version of the package for Focal.

[Robie Basak (racb)]

I see the following in the diff. Is this an intended change? I don't see any mention of it in the changelog or in this bug:

--- a/DistUpgrade/apt_btrfs_snapshot.py
+++ b/DistUpgrade/apt_btrfs_snapshot.py
@@ -73,11 +73,11 @@ class Fstab(list):
         super(Fstab, self).__init__()

         with open(fstab) as fstab_file:
- for line in (l.strip() for l in fstab_file):
- if line == "" or line.startswith("#"):
+ for stripped_line in (line.strip() for line in fstab_file):
+ if stripped_line == "" or stripped_line.startswith("#"):
                     continue
                 try:
- entry = FstabEntry.from_line(line)
+ entry = FstabEntry.from_line(stripped_line)

[Robie Basak (racb)]

Hello Matthew, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.40 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

(FTR, comment 4 was addressed by a new upload that removed that change; see #ubuntu-devel for the debate)

[Matthew Ruffell (mruffell)]

The debate mentioned in comment 6 has been reproduced here:

https://paste.ubuntu.com/p/Hq4zjk6Gxn/

It has been resolved, with the removal of the DistUpgrade/apt_btrfs_snapshot.py hunk that was brought in via prebuild.sh

https://paste.ubuntu.com/p/7xsQXtCMYc/

No further action necessary for comment 4 and 6.

[Matthew Ruffell (mruffell)]

Performing verification for Focal

I started three Bionic VMs and set up Ubuntu Pro like so:

- Bionic
- Bionic + fips
- Bionic + fips-updates

On each of these VMs I ran

$ sudo do-release-upgrade --proposed

This used ubuntu-release-upgrader 1:20.04.40 from --proposed, and started the
upgrade process.

The plain Bionic VM upgraded successfully. Upgrade log:

https://paste.ubuntu.com/p/j7V33YHHV8/

Looking at the fips related entries, we see:

2023-01-26 01:45:09,918 DEBUG _fipsLibgcryptDivert
2023-01-26 01:45:09,919 DEBUG System is not fips enabled, no dpkg-divert needed
...
2023-01-26 02:01:49,401 DEBUG _fipsLibgcryptHmacCleanup
2023-01-26 02:01:49,402 DEBUG System is not fips enabled, no need for hmac tidy

The Bionic + fips VM upgraded successfully. Upgrade log:

https://paste.ubuntu.com/p/Y3tqXtN97W/

Looking at the fips related entries, we see:

2023-01-26 01:56:58,731 DEBUG _fipsLibgcryptDivert
2023-01-26 01:56:58,732 DEBUG Broken libgcrypt20-hmac will be installed, likely due to using certified fips pocket instead of fips-updates. Installing dpkg-divert for incorrect placement of /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the correct /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
...
2023-01-26 02:17:27,649 DEBUG _fipsLibgcryptHmacCleanup
2023-01-26 02:17:27,649 DEBUG Removing old hmac file from Bionic version

Looking at the hmac file locations:

$ ll /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
ls: cannot access '/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac': No such file or directory
$ ll /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
-rw-r--r-- 1 root root 126 May 26 2021 /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
$ dpkg-divert --list
...
local diversion of /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
$ apt-cache policy libgcrypt20-hmac | grep Installed
  Installed: 1.8.5-5ubuntu1.fips.1.4

The quirk worked as intended and the upgrade was successful.

The Bionic + fips-updates VM upgraded successfully. Upgrade log:

https://paste.ubuntu.com/p/cDVyPsFRKB/

2023-01-26 01:57:51,662 DEBUG _fipsLibgcryptDivert
2023-01-26 01:57:51,663 DEBUG Fixed libgcrypt20-hmac will be installed as part of upgrade process, no need for dpkg-divert
...
2023-01-26 03:23:16,764 DEBUG _fipsLibgcryptHmacCleanup
2023-01-26 03:23:16,764 DEBUG Fixed libgcrypt20-hmac was installed as part of upgrade process, no need for hmac file cleanup

$ ll /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
ls: cannot access '/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac': No such file or directory
$ ll /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
-rw-r--r-- 1 root root 126 Apr 15 2022 /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
$ dpkg-divert --list
...
$ apt-cache policy libgcrypt20-hmac | grep Installed
  Installed: 1.8.5-5ubuntu1.fips.1.7

The quirk noticed the fixed libgcrypt20-hmac package was installed during the upgrade, and did not set up any unnecessary dpkg-diverts.

In all cases, the quirk functioned as intended, and the upgrades from Bionic to Focal were successful. The ubuntu-release-upgrader package 1:20.04.40 from --proposed fixes the issue, happy to mark verified for Focal.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.40

---------------
ubuntu-release-upgrader (1:20.04.40) focal; urgency=medium

  [ Matthew Ruffell ]
  * DistUpgrade/DistUpgradeQuirks.py: Add quirk to handle fips specific
    .libgcrypt.so.20.hmac file from libgcrypt20-hmac being placed in
    /lib instead of /usr/lib, breaking upgrades for fips enabled systems.
    (LP: #1982534)

  [ Brian Murray ]
  * Update mirrors, demotions, and translations.

 -- Matthew Ruffell <email address hidden> Mon, 21 Nov 2022 11:31:53 +1300