| 2023-01-10 04:36:29 |
Matthew Ruffell |
description |
[Impact]
When upgrading a Bionic fips (NOT fips-updates) system to Focal fips, the upgrade gets stuck generating the initramfs with the following message:
update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic
Failed to copy HMAC file "/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac".
E: /usr/share/initramfs-tools/hooks/fips-libgcrypt failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-84-generic with 1.
dpkg: error processing package ubuntu-fips (--configure):
installed ubuntu-fips package post-installation script subprocess returned error exit status 1
This happens because we are trying to copy /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the initramfs, but because of a bug, it is actually installed to /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac.
Most normal Focal systems are not affected, due to being usrmerged [1], where /lib is a symlink to /usr/lib.
[1] https://wiki.debian.org/UsrMerge
But when we upgrade from Bionic, usrmerge is not performed.
We have resolved this bug in two new package releases for Focal, in fips-updates:
- ubuntu-fips (1.2.4+updates1)
- libgcrypt20 (1.8.5-5ubuntu1.fips.1.5)
This was a part of bug 1944403, which is private due to having debdiffs for FIPS packages attached to the bug. Ask mruffell for access if you wish to view the bug.
We changed /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be placed in /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac, and we changed the initramfs hook script to look for existence in both /usr/lib and /lib.
The problem is, libgcrypt20 (1.8.5-5ubuntu1.fips.1.5) did not make the current re-certification round and Canonical have already received a certificate for libgcrypt20 (1.8.5-5ubuntu1.fips.1.4) stating fips compliance, thus we cannot get 1.8.5-5ubuntu1.fips.1.5 certified in this current round.
The next certification round may take place in a year or two, but we require a solution for users who wish to upgrade from Bionic fips to Focal fips, and those users cannot enable fips-updates for strict compliance reasons.
The workaround is to add a dpkg-divert, somewhat of the form of:
$ [ ! -L /lib ] && sudo dpkg-divert --local --rename --add --divert /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
I have implemented this as a upgrade quirk in DistUpgradeQuirks.py, in the function _fipsLibgcryptDivert(), which checks for the presence of libgcrypt20-hmac and checking to see if the broken fips certified package would be installed, before adding the dpkg-divert.
Note, I did not use the --rename option in the quirk, as we require /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be left in place while we are performing the upgrade, as the libgcrypt self-test will fail if the file is removed before we install the Focal libgcrypt package. The error is very noisy as it is printed each time a program is invoked that uses libgcrypt, e.g. apt, dpkg, and it could prevent the upgrade from completing successfully.
libgcrypt selftest: binary (0): No such file or directory (/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)
Instead, we leave /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac in place until the very end of the upgrade, where we remove it in the function _fipsLibgcryptHmacCleanup().
[Testcase]
There are three upgrades that need to be tested to ensure things work as intended:
- Bionic
- Bionic fips
- Bionic fips-updates
Systems need to be attached to a UA token, and fips or fips-updates enabled and installed, before upgrade starts.
You can use do-release-upgrade to perform the upgrade.
The patch can be found here:
https://paste.ubuntu.com/p/FgrnsRfZPt/
You can test the upgrade before the merge to ubuntu-release-upgrader by:
$ sudo -s
# vim ~/quirk.patch
Stick the contents of the pastebin in the file.
# apt update
# apt upgrade
# do-release-upgrade
At the first screen where you are asked to start another ssh server, hit ctrl-c
and then press x to exit screen.
We need to patch the quirk file we just downloaded.
# cd /tmp
# cd ubuntu-release-upgrader-<RANDOM LETTERS>
# patch -p1 < ~/quirk.patch
Hopefully it applies cleanly.
We can restart do-release-upgrade with our patched copy with:
# ./focal
From there, all upgrades should succeed cleanly.
Ideally we would also test upgrading to Jammy from the resulting Focal install, to ensure usrmerge works as intended, and that /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac was cleaned up correctly.
[Where problems could occur]
We are adding a quirk to Bionic systems upgrading to Focal only. This quirk is not intended to be used on any platform other than Bionic->Focal, and thus only targets the Focal branch of ubuntu-release-upgrader.
We check to make sure that libgcrypt20-hmac is installed, and if the package is not installed, we exit from the quirk. This way we ensure we only apply the quirk to FIPS enabled systems.
We also exit from the quirk if fips-updates is being used, and the fixed libgcrypt20-hmac package is to be installed as part of the upgrade.
On cleanup, we take great care to ensure that we only remove the Bionic /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file, and leave the Focal /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file well alone. We ensure the system is not usrmerged, and check the hashes of both files to make sure they do not match before removing the Bionic hmac file.
If a regression were to occur, it could prevent systems from upgrading correctly, and would have a significant impact to users before it would be fixed.
I believe I have narrowed the risk down to the small pool of fips users that require this quirk to be able to upgrade successfully. |
[Impact]
When upgrading a Bionic fips (NOT fips-updates) system to Focal fips, the upgrade gets stuck generating the initramfs with the following message:
update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic
Failed to copy HMAC file "/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac".
E: /usr/share/initramfs-tools/hooks/fips-libgcrypt failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-84-generic with 1.
dpkg: error processing package ubuntu-fips (--configure):
installed ubuntu-fips package post-installation script subprocess returned error exit status 1
This happens because we are trying to copy /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the initramfs, but because of a bug, it is actually installed to /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac.
Most normal Focal systems are not affected, due to being usrmerged [1], where /lib is a symlink to /usr/lib.
[1] https://wiki.debian.org/UsrMerge
But when we upgrade from Bionic, usrmerge is not performed.
We have resolved this bug in two new package releases for Focal, in fips-updates:
- ubuntu-fips (1.2.4+updates1)
- libgcrypt20 (1.8.5-5ubuntu1.fips.1.5)
This was a part of bug 1944403, which is private due to having debdiffs for FIPS packages attached to the bug. Ask mruffell for access if you wish to view the bug.
We changed /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be placed in /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac, and we changed the initramfs hook script to look for existence in both /usr/lib and /lib.
The problem is, libgcrypt20 (1.8.5-5ubuntu1.fips.1.5) did not make the current re-certification round and Canonical have already received a certificate for libgcrypt20 (1.8.5-5ubuntu1.fips.1.4) stating fips compliance, thus we cannot get 1.8.5-5ubuntu1.fips.1.5 certified in this current round.
The next certification round may take place in a year or two, but we require a solution for users who wish to upgrade from Bionic fips to Focal fips, and those users cannot enable fips-updates for strict compliance reasons.
The workaround is to add a dpkg-divert, somewhat of the form of:
$ [ ! -L /lib ] && sudo dpkg-divert --local --rename --add --divert /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac
I have implemented this as a upgrade quirk in DistUpgradeQuirks.py, in the function _fipsLibgcryptDivert(), which checks for the presence of libgcrypt20-hmac and checking to see if the broken fips certified package would be installed, before adding the dpkg-divert.
Note, I did not use the --rename option in the quirk, as we require /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be left in place while we are performing the upgrade, as the libgcrypt self-test will fail if the file is removed before we install the Focal libgcrypt package. The error is very noisy as it is printed each time a program is invoked that uses libgcrypt, e.g. apt, dpkg, and it could prevent the upgrade from completing successfully.
libgcrypt selftest: binary (0): No such file or directory (/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)
Instead, we leave /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac in place until the very end of the upgrade, where we remove it in the function _fipsLibgcryptHmacCleanup().
[Testcase]
There are three upgrades that need to be tested to ensure things work as intended:
- Bionic
- Bionic fips
- Bionic fips-updates
Systems need to be attached to a UA token, and fips or fips-updates enabled and installed, before upgrade starts.
You can use do-release-upgrade to perform the upgrade.
The patch can be found here:
https://paste.ubuntu.com/p/FgrnsRfZPt/
You can test the upgrade before the merge to ubuntu-release-upgrader by:
$ sudo -s
# vim ~/quirk.patch
Stick the contents of the pastebin in the file.
# apt update
# apt upgrade
# do-release-upgrade
At the first screen where you are asked to start another ssh server, hit ctrl-c
and then press x to exit screen.
We need to patch the quirk file we just downloaded.
# cd /tmp
# cd ubuntu-release-upgrader-<RANDOM LETTERS>
# patch -p1 < ~/quirk.patch
Hopefully it applies cleanly.
We can restart do-release-upgrade with our patched copy with:
# ./focal
From there, all upgrades should succeed cleanly.
Ideally we would also test upgrading to Jammy from the resulting Focal install, to ensure usrmerge works as intended, and that /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac was cleaned up correctly.
[Where problems could occur]
We are adding a quirk to Bionic systems upgrading to Focal only. This quirk is not intended to be used on any platform other than Bionic->Focal, and thus only targets the Focal branch of ubuntu-release-upgrader.
We check to make sure that libgcrypt20-hmac is installed, and if the package is not installed, we exit from the quirk. This way we ensure we only apply the quirk to FIPS enabled systems.
We also exit from the quirk if fips-updates is being used, and the fixed libgcrypt20-hmac package is to be installed as part of the upgrade.
On cleanup, we take great care to ensure that we only remove the Bionic /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file, and leave the Focal /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file well alone. We ensure the system is not usrmerged, and check to ensure the files are not symlinks before removing the Bionic hmac file.
If a regression were to occur, it could prevent systems from upgrading correctly, and would have a significant impact to users before it would be fixed.
I believe I have narrowed the risk down to the small pool of fips users that require this quirk to be able to upgrade successfully. |
|
