Apt/Grub issue with 22.04 Upgrade (Missing libcrypto.so.1.1)

Bug #1969353 reported by Batwam
52
This bug affects 9 people
Affects Status Importance Assigned to Milestone
grub-customizer (Ubuntu)
Won't Fix
Undecided
Unassigned
Focal
Confirmed
Undecided
Unassigned
Impish
Confirmed
Undecided
Unassigned
ubuntu-release-upgrader (Ubuntu)
Won't Fix
Undecided
Unassigned
Focal
Confirmed
Undecided
Unassigned
Impish
Confirmed
Undecided
Unassigned

Bug Description

I would like to report an issue with the final phase of the upgrade to 22.04 (the final package cleanup) where it tries to update grub after removing some older kernels but fails due to a missing file (libcrypto.so.1.1) - See details in the log attached.

I was unable to fix this using apt alone since it abolutely wants to fix this. I identified that the missing file was part of the libssl package. I added Impish-Security back into the sources.list but I still couldn't install libssl1.1 as apt wanted to update grub first for some reason.

For anyone with this issue, I managed to fix it eventually by downloading the libssl deb manually and running 'sudo apt --fix-broken install'.

I don't know for sure where the issue comes from but this was a farly recent install of Ubuntu 21.10. I can see that libssl existed up to Impish only so it's either a temporary issue and it will be added between now and the upgrade date, or the package has been removed and some other people may face the same problem.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: ubuntu-release-upgrader-core 1:22.04.9
ProcVersionSignature: Ubuntu 5.15.0-25.25-generic 5.15.30
Uname: Linux 5.15.0-25-generic x86_64
ApportVersion: 2.20.11-0ubuntu82
Architecture: amd64
CasperMD5CheckResult: pass
CrashDB: ubuntu
CrashReports:
 640:1000:123:6404482:2022-04-18 14:59:09.480963945 +0800:2022-04-18 15:37:28.386895744 +0800:/var/crash/_usr_bin_gnome-control-center.1000.crash
 600:0:123:801423:2022-04-18 15:23:35.872834781 +0800:2022-04-18 15:37:28.402892412 +0800:/var/crash/linux-image-5.13.0-39-generic.0.crash
 640:0:123:395731:2022-04-18 16:31:42.762021782 +0800:2022-04-18 16:31:42.434015345 +0800:/var/crash/_bin_brltty.0.crash
CurrentDesktop: ubuntu:GNOME
Date: Mon Apr 18 17:25:32 2022
InstallationDate: Installed on 2022-04-06 (11 days ago)
InstallationMedia: Ubuntu 21.10 "Impish Indri" - Release amd64 (20211012)
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
Symptom: release-upgrade
UpgradeStatus: Upgraded to jammy on 2022-04-18 (0 days ago)
VarLogDistupgradeXorgFixuplog:
 INFO:root:/usr/bin/do-release-upgrade running
 INFO:root:No xorg.conf, exiting

Revision history for this message
Batwam (batwam) wrote :
Batwam (batwam)
summary: - Apt/Grub issue with 22.04 Upgrade (Missing Dependency)
+ Apt/Grub issue with 22.04 Upgrade (Missing libcrypto.so.1.1)
Revision history for this message
Batwam (batwam) wrote :

For clarity, when I talk about libssl being removed above, I am referring specifically to libssl1.1. Looking at the logs (and various posts online), it looks like there is a move from libssl1.1 to libssl3 as part of this upgrade which seems to be the source of the issue.

Revision history for this message
Batwam (batwam) wrote :

Upon further inspection of the log, I realised that the issue occurs when /etc/grub.d/bin/grubcfg_proxy is being run but this file is not part of grub itself. Looking through the package contents, it looks like grubcfg-cfg is included in grub-customizer which is installed on my system so, this is perhaps actually and issue with the grub-customizer package?

It looks like it is not the first time this package has this kind of compatibility issues: https://askubuntu.com/questions/1168054/how-to-fix-kernel-issue-after-upgrading-from-18-10-to-19-04

Revision history for this message
Simon Chopin (schopin) wrote :

My first guess, just the askubuntu link, was that it was a leftover from an old version of the grub-customizer package that hasn't been removed as it is in /etc and subject to conffile handling, but I'm not so sure that's entirely right, as I'd gathered that the old version was from before Disco and this is a fresh Impish install.

Revision history for this message
Simon Chopin (schopin) wrote :

I looked into the details, and my conclusions are that the grub-customizer package has a completely broken removal/upgrade story.

The package only installs some binaries, without any custom maintainer scripts or fancy things. Once you run the main program, it goes and modifies the /etc/grub.d scripts automatically for you to customize them as specified. That part already feels weird to me, but wouldn't necessarily cause problems.

However, there are some customizations that are apparently not doable by simply editing the existing configuration scripts. For those, the program moves all existing conf scripts out of the way, and instead installs "proxy scripts" written in a custom language that basically filter the output of the original scripts. The language interpreter is a standard ELF binary with standard dynamic linking, which seems to need the whole Qt stack for some reason. Said interpreter is copied to /etc/grub.d/bin/grubcfg_proxy, presumably so that the system keeps working even if the user uninstalls grub-customizer.

That logic is actually broken, since removing the grub-customizer package might lead to the removal of its dependencies, e.g. the Qt stack, leading to a broken binary.

My suggestion for the path forward would be to patch the application to have /etc/grub.d/bin/grubcfg_proxy be a shell script that invokes the dpkg-managed proxy if it's available, else prints out a warning and executes the original script. We couple that with a post-rm maintainer script that ensures that the /etc-installed script is the one from the current version of the package. We'd need to SRU this to Impish, Focal, and Jammy.

That wouldn't fix every issue with the package, notably leaving the purge situation still broken, but at least that'd fix the problem at hand.

Thoughts?

Revision history for this message
Simon Chopin (schopin) wrote (last edit ):

In the meantime, you can do this to fix your system:

sudo cp /usr/lib/grub-customizer/grubcfg-proxy /etc/grub.d/bin/grubcfg_proxy

EDIT: see https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1969353/comments/14 for a new workaround since the package has been completely removed from the archive

Revision history for this message
Batwam (batwam) wrote :

Bonjour Simon, I actually fixed my system at the time by downloading the libssl1.1 deb from Impish but your solution would have worked as well. It looks like this grubcfg_proxy file gets copied when running grub-customizer so, presumably, since I didn’t run the tool after the upgrade, it still had the old version of grubcfg_proxy which relied on libssl1.1.

It looks like if the _proxy files in the grub.d directory (10_linux_proxy,…) linked to /usr/lib/grub-customizer/grubcfg-proxy rather than a copy of the file (or perhaps simpler if grubcfg_proxy was a symlink back to /usr/lib/…), this wouldn’t have been an issue?

This said, I also agree that, more boradly, if the _proxy files are left after grub-customizer is deleted, this is potentially a recipe for disaster…

Simon Chopin (schopin)
tags: added: fr-2262
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub-customizer (Ubuntu Focal):
status: New → Confirmed
Changed in grub-customizer (Ubuntu Impish):
status: New → Confirmed
Changed in grub-customizer (Ubuntu):
status: New → Confirmed
Changed in ubuntu-release-upgrader (Ubuntu Focal):
status: New → Confirmed
Changed in ubuntu-release-upgrader (Ubuntu Impish):
status: New → Confirmed
Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
Revision history for this message
Simon Chopin (schopin) wrote :

I just realized that my workaround only worked as long as the grub-customizer was in the archive. Since we've removed it due to the above issues, the workaround would probably be something like this (completely untested):

orig_dir=/etc/grub.d/proxifiedScripts
main_dir=/etc/grub.d
backup_dir=/etc/grub.d/proxyscript_backup

mkdir -p $backup_dir
for script in $(ls $orig_dir); do
    # we're assuming there's only one proxy script for each proxied file
    proxy_script=$(ls ${main_dir}/*_${script}_proxy)
    # back up the "proxy script"
    cp $main_dir/$proxy_script $backup_dir
    # overwrite it with the original shell script
    cp $orig_dir/$script /main_dir
done

Revision history for this message
Batwam (batwam) wrote :

the final "/main_dir" looks like a typo. also, the first cp would fail as $proxy_script already includes the full path. One problem I see is that, for me at least, the files in the proxifiedScripts folders have been stripped of their numbers ("10_linux" is called "linux" for instance). so before copying the file back, you would need to take the script name and strip the _proxy part to find the original name:

#!/bin/bash
main_dir=/etc/grub.d
orig_dir=$main_dir/proxifiedScripts
backup_dir=$main_dir/proxyscript_backup

mkdir -p $backup_dir
for script in $(ls $orig_dir); do
    # we're assuming there's only one proxy script for each proxied file
    proxy_script=$(ls ${main_dir}/*_"${script}"_proxy)
    new_script=$(echo "$proxy_script" | sed 's/_proxy//g')
    # back up the "proxy script"
    mv "$proxy_script" $backup_dir
    # overwrite it with the original shell script if it doesn't already exist
    if [ ! -f "$new_script" ]; then
       cp $orig_dir/"$script" "$new_script"
    else
        echo "file already exists"
    fi
done

Also, if 10_linux has been copied during the upgrade, chances are that the files in the proxy folder is an older version so I wouldn't really want the new files to be overwritten with the old ones so, a simple sudo rm /etc/grub.d/*_proxy works just as well for me to prevent the grub failure which I experienced post upgrade. People can always fire up grub-customizer again and regenerate the files if needed.

Alternatively, could it be as simple as copying /etc/grub.d/bin/grubcfg_proxy as part of the package install/update?

Revision history for this message
Simon Chopin (schopin) wrote : Re: [Bug 1969353] Re: Apt/Grub issue with 22.04 Upgrade (Missing libcrypto.so.1.1)

All good points there, just one thing:

Quoting Batwam (2022-04-22 17:34:37)
> Alternatively, could it be as simple as copying
> /etc/grub.d/bin/grubcfg_proxy as part of the package install/update?

You still have a broken system on package removal.

Revision history for this message
Batwam (batwam) wrote :

yes, you are right, espectially considering the point you made about the underlying dependancies this script relies on (libssl1.1 being one) which is what caused my Impish->Jammy upgrade issue in the first place.

Revision history for this message
Batwam (batwam) wrote :

By the way, just to be clear, it’s not just the package removal which is a problem. Upgrade is also a problem as per the original but report as the file in the bin folder isn’t getting updated while libssl is changed from 1.1 to 3. I’m just raising this as more people are going to be upgrading their system in the coming weeks and might be faced with the same problem I had.

Revision history for this message
Valentin Vakrilov (vvakrilov) wrote :

Hi!
I've got (I think) the same problem here:

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
  linux-image-5.13.0-39-generic
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 10,3 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 223072 files and directories currently installed.)
Removing linux-image-5.13.0-39-generic (5.13.0-39.44) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.13.0-39-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.0-27-generic
Found initrd image: /boot/initrd.img-5.15.0-27-generic
Found linux image: /boot/vmlinuz-5.13.0-40-generic
Found initrd image: /boot/initrd.img-5.13.0-40-generic
/etc/grub.d/10_linux_proxy: 3: /etc/grub.d/bin/grubcfg_proxy: not found
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-5.13.0-39-generic (--remove):
 installed linux-image-5.13.0-39-generic package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 linux-image-5.13.0-39-generic
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)

Revision history for this message
Michal Petřek (ersaf) wrote (last edit ):

Hi all,
found out, that grub customizer provides backup of all the files before its install in /etc/grub.d/backup. One workaround is simply just follow RESTORE_INSTRUCTIONS placed there. Obviously there is a problem, that all changes made to grub after customizer install will be lost, but after I realized, what a mess this package made, I refuse to use it again.
Not sure, if this is possible after running release upgrade. I landed here thanks to release notes, so I am doing this as a precaution.

EDIT: This is really NOT the way. I messed up my grub by overwriting /boot/grub and I am so stupid, that this is the only folder I forget to backup before these changes. I will have some fun today...

Revision history for this message
Batwam (batwam) wrote :

Hi Michael, my understanding is that proxified scripts are backed up in proxifiedScripts. If you look at the previous comments, you will find a short script I proposes which removes the proxy files and put the original ones back in (unless you saved new ones during upgrade). You might find the files you need there.

Revision history for this message
Michal Petřek (ersaf) wrote :

I saw your script but I wanted just to try it my way. When I found out that grub is broken and couldn't find some fast and straightforward solution, I just made a clean install. All custom config backed up and few apps I use was installed in no time. Anyway thank you for help.

Revision history for this message
Batwam (batwam) wrote :

No worries.

There is actually a fairly easy solution which doesn’t involve relying on the scripts above and that’s the one from the very first post which is in in brief:
- download and install libssl1.1 from the Impish archive (https://packages.ubuntu.com/impish/libssl1.1). It doesn’t look like there is anything preventing people from having both libssl1.1 and libssl3.
- install it (do ‘sudo dpkg -i package.deb’
- fix apt by running ‘ sudo apt --fix-broken install'.

Revision history for this message
Mario (mr-gmzc) wrote :

Thanks for the solution Batwam, can confirm it worked with no issues

Steve Langasek (vorlon)
Changed in grub-customizer (Ubuntu):
status: Confirmed → Won't Fix
Changed in ubuntu-release-upgrader (Ubuntu):
status: Confirmed → Won't Fix
Revision history for this message
Davide Pecis (peach1200) wrote :

Newbie Ubuntu user here, didn´t have any clue of what to do.
Batwam solution worked like a charm and it was very easy. Thank you very much!

Revision history for this message
Batwam (batwam) wrote :

glad it helped you Davide.

Im really not sure why the Ubuntu team decided to make this a "Won't Fix" as the solution, it would have been great to get some sort of explaination as the package clearly isn't bahaving the way it should.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers