usrmerge error after upgrading from 20.10 to 21.04

[Mathieu Austruy]

[Impact]

 * Starting with Ubuntu Hirsute, the 'usrmerge' package is pulled into the default installation, doing the merging of /{bin,sbin,lib}/ and /usr/{bin,sbin,lib} on upgrade. If there are left-over generated files it will fail with an error like this:

Setting up usrmerge (24ubuntu3) ...

FATAL ERROR:
Both /lib/udev/hwdb.bin and /usr/lib/udev/hwdb.bin exist.

The upgrade has completed but there were errors during the upgrade
process.

 * We want a fix in Hirsute+ that clears the old location (e.g. /lib/udev/hwdb.bin) if the file has already been re-generated at the new location (e.g. /usr/lib/udev/hwdb.bin).

[Test Plan]

Prepare a recent, but non-usrmerge'd system (e.g. launch Bionic and upgrade to Focal then Hirsute):
$ lxc launch ubuntu:bionic usrmerge
# do-release-upgrade
[...]
# reboot
# ls -la / # to make sure that /lib is not (yet) a link to /usr/lib

Simulate duplicated generated files & upgrade:
# mkdir -p /usr/lib/udev && cp /lib/udev/hwdb.bin /usr/lib/udev/
# vim /etc/update-manager/release-upgrades # change to "Prompt=normal"
# do-release-upgrade
[...]
=> Upgrade fails, press "C" to cancel bug report
=> "The upgrade has completed but there were errors during the upgrade process."

Verify that the script still fails:
# /usr/lib/usrmerge/convert-usrmerge
FATAL ERROR:
Both /lib/udev/hwdb.bin and /usr/lib/udev/hwdb.bin exist.

Install relevant usrmerge package:
# wget https://launchpad.net/ubuntu/+archive/primary/+files/usrmerge_VER_all.deb
# dpkg -i usrmerge_VER_all.deb
=> The system has been successfully converted.
# /usr/lib/usrmerge/convert-usrmerge
=> The system has been successfully converted.
# echo $?
0

[Where problems could occur]

 * The upload modifies the convert-usrmerge script, adding special handling for certain, generated files.
 * If something goes wrong, it could delete the wrong state of udev's hwdb which would be re-generated by systemd-hwdb-update.service
 * It could also interrupt the upgrade processes on installation of the usrmerge package, asking the user for manual action (e.g. solving some file conflict and re-running /usr/lib/usrmerge/convert-usrmerge)

[Other Info]

 * The patch was sent to Debian: https://bugs.debian.org/997866

=== Original Description ===

I got the following error after upgrading from 20.10 to 21.04:

Setting up usrmerge (24ubuntu3) ...

FATAL ERROR:
Both /lib/udev/hwdb.bin and /usr/lib/udev/hwdb.bin exist.

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.

dpkg: error processing package usrmerge (--configure):
 installed usrmerge package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 usrmerge

Running "sudo dpkg --configure -a", "sudo apt -f install" or "sudo /usr/lib/usrmerge/convert-usrmerge" give exactly the same error.

[Mathieu Austruy (maustruy)]

May be related to https://bugs.launchpad.net/ubuntu/+source/usrmerge/+bug/1924850

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1930573/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[abdullahc (sneetsher)]

`hwdb.bin` is a generated file. I did check it in Ubuntu 20.04, `/lib/udev/hwdb.bin` and `/usr/lib/udev/hwdb.bin` are identical copy, so no need to be merged.

A quick fix would be deleting one of them:

```sudo rm /lib/udev/hwdb.bin```

Then fix apt using: `sudo apt install -f`

usrmerge should probably make same action.

[Brian Murray (brian-murray)]

Hello Mathieu, or anyone else affected,

Accepted usrmerge into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/usrmerge/25ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Mathieu, or anyone else affected,

Accepted usrmerge into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/usrmerge/24ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lukas Märdian (slyon)]

Thank you Brian for accepting it into impish-proposed!

I've tested usrmerge 25ubuntu1.1 from impish-proposed, all good:

$ lxc launch ubuntu:bionic usrmerge
$ lxc exec usrmerge bash
# do-release-upgrade
[...]
# reboot
# ls -l /
total 88
drwxr-xr-x 2 root root 12288 Oct 27 13:39 bin
drwxr-xr-x 2 root root 4096 Oct 21 15:54 boot
drwxr-xr-x 8 root root 500 Oct 27 13:39 dev
drwxr-xr-x 94 root root 12288 Oct 27 13:39 etc
drwxr-xr-x 3 root root 4096 Oct 27 13:30 home
drwxr-xr-x 21 root root 4096 Oct 27 13:35 lib
drwxr-xr-x 2 root root 4096 Oct 27 13:31 lib64
drwxr-xr-x 2 root root 4096 Oct 21 15:48 media
drwxr-xr-x 2 root root 4096 Oct 21 15:48 mnt
drwxr-xr-x 2 root root 4096 Oct 21 15:48 opt
dr-xr-xr-x 512 nobody nogroup 0 Oct 27 13:39 proc
drwx------ 4 root root 4096 Oct 27 13:39 root
drwxr-xr-x 20 root root 640 Oct 27 13:39 run
drwxr-xr-x 2 root root 12288 Oct 27 13:39 sbin
drwxr-xr-x 6 root root 4096 Oct 27 13:35 snap
drwxr-xr-x 2 root root 4096 Oct 21 15:48 srv
dr-xr-xr-x 13 nobody nogroup 0 Oct 27 13:39 sys
drwxrwxrwt 10 root root 4096 Oct 27 13:39 tmp
drwxr-xr-x 12 root root 4096 Oct 27 13:33 usr
drwxr-xr-x 13 root root 4096 Oct 21 15:52 var
# mkdir -p /usr/lib/udev && cp /lib/udev/hwdb.bin /usr/lib/udev/
# sed -i 's/Prompt=lts/Prompt=normal/' /etc/update-manager/release-upgrades
# do-release-upgrade
[...]
# /usr/lib/usrmerge/convert-usrmerge
ionice: ioprio_set failed: Operation not permitted
chrt: failed to set pid 44628's policy: Operation not permitted

FATAL ERROR:
Both /lib/udev/hwdb.bin and /usr/lib/udev/hwdb.bin exist.

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.

# echo $?
1
# wget https://launchpad.net/ubuntu/+archive/primary/+files/usrmerge_25ubuntu1.1_all.deb
# dpkg -i usrmerge_25ubuntu1.1_all.deb
(Reading database ... 35984 files and directories currently installed.)
Preparing to unpack usrmerge_25ubuntu1.1_all.deb ...
Unpacking usrmerge (25ubuntu1.1) over (24ubuntu3) ...
Setting up usrmerge (25ubuntu1.1) ...
Smartmatch is experimental at /usr/lib/usrmerge/convert-usrmerge line 172.
ionice: ioprio_set failed: Operation not permitted
chrt: failed to set pid 44665's policy: Operation not permitted
The system has been successfully converted.
# /usr/lib/usrmerge/convert-usrmerge
Smartmatch is experimental at /usr/lib/usrmerge/convert-usrmerge line 172.
ionice: ioprio_set failed: Operation not permitted
chrt: failed to set pid 44812's policy: Operation not permitted
The system has been successfully converted.
# echo $?
0
# ls -la /
total 64
drwxr-xr-x 18 root root 4096 Oct 27 13:53 .
drwxr-xr-x 18 root root 4096 Oct 27 13:53 ..
lrwxrwxrwx 1 root root 7 Oct 27 13:53 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Oct 21 15:54 boot
drwxr-xr-x 8 root root 500 Oct 27 13:39 dev
drwxr-xr-x 101 root root 12288 Oct 27 13:53 etc
drwxr-xr-x 3 root root 4096 Oct 27 13:30 ...

[Lukas Märdian (slyon)]

Thank you Brian for accepting it into hirsute-proposed!

I've tested usrmerge 24ubuntu3.1 from hirsute-proposed, all good:

$ lxc launch ubuntu:bionic usrmerge
$ lxc exec usrmerge bash
# do-release-upgrade
[...]
# reboot
# ls -la /
total 96
drwxr-xr-x 22 root root 4096 Oct 21 15:54 .
drwxr-xr-x 22 root root 4096 Oct 21 15:54 ..
drwxr-xr-x 2 root root 12288 Oct 27 13:11 bin
drwxr-xr-x 2 root root 4096 Oct 21 15:54 boot
drwxr-xr-x 8 root root 500 Oct 27 13:12 dev
drwxr-xr-x 94 root root 12288 Oct 27 13:12 etc
drwxr-xr-x 3 root root 4096 Oct 27 13:03 home
drwxr-xr-x 21 root root 4096 Oct 27 13:08 lib
drwxr-xr-x 2 root root 4096 Oct 27 13:05 lib64
drwxr-xr-x 2 root root 4096 Oct 21 15:48 media
drwxr-xr-x 2 root root 4096 Oct 21 15:48 mnt
drwxr-xr-x 2 root root 4096 Oct 21 15:48 opt
dr-xr-xr-x 502 nobody nogroup 0 Oct 27 13:12 proc
drwx------ 4 root root 4096 Oct 27 13:12 root
drwxr-xr-x 22 root root 700 Oct 27 13:12 run
drwxr-xr-x 2 root root 12288 Oct 27 13:11 sbin
drwxr-xr-x 6 root root 4096 Oct 27 13:08 snap
drwxr-xr-x 2 root root 4096 Oct 21 15:48 srv
dr-xr-xr-x 13 nobody nogroup 0 Oct 27 13:12 sys
drwxrwxrwt 11 root root 4096 Oct 27 13:12 tmp
drwxr-xr-x 12 root root 4096 Oct 27 13:07 usr
drwxr-xr-x 13 root root 4096 Oct 21 15:52 var
# mkdir -p /usr/lib/udev && cp /lib/udev/hwdb.bin /usr/lib/udev/
# sed -i 's/Prompt=lts/Prompt=normal/' /etc/update-manager/release-upgrades
# do-release-upgrade
[...]
Errors were encountered while processing:
 usrmerge
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (238.1 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C): C

Could not install the upgrades

The upgrade has aborted. Your system could be in an unusable state. A
recovery will run now (dpkg --configure -a).

Setting up usrmerge (24ubuntu3) ...
ionice: ioprio_set failed: Operation not permitted
chrt: failed to set pid 44214's policy: Operation not permitted

FATAL ERROR:
Both /lib/udev/hwdb.bin and /usr/lib/udev/hwdb.bin exist.

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.

dpkg: error processing package usrmerge (--configure):
 installed usrmerge package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 usrmerge

Upgrade complete

The upgrade has completed but there were errors during the upgrade
process.

To continue please press [ENTER]
=> [C] [ENTER] [x]
# /usr/lib/usrmerge/convert-usrmerge
ionice: ioprio_set failed: Operation not permitted
chrt: failed to set pid 44...

[Launchpad Janitor (janitor)]

This bug was fixed in the package usrmerge - 25ubuntu2

---------------
usrmerge (25ubuntu2) jammy; urgency=medium

  * Clear generated files if they have already been re-generated (LP: #1930573)

 -- Lukas Märdian <email address hidden> Tue, 26 Oct 2021 11:12:18 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package usrmerge - 25ubuntu1.1

---------------
usrmerge (25ubuntu1.1) impish; urgency=medium

  * Clear generated files if they have already been re-generated (LP: #1930573)

 -- Lukas Märdian <email address hidden> Tue, 26 Oct 2021 12:42:55 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for usrmerge has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package usrmerge - 24ubuntu3.1

---------------
usrmerge (24ubuntu3.1) hirsute; urgency=medium

  * Clear generated files if they have already been re-generated (LP: #1930573)

 -- Lukas Märdian <email address hidden> Tue, 26 Oct 2021 12:47:37 +0200

[Rok Kvedr (kvedr)]

After upgrading from 20.04 to 22.04.1. problem showed up with usrmerge. I have manually running

------------------------------------------
sudo /usr/lib/usrmerge/convert-usrmerge
------------------------------------------

repaired few errors like

--------------------------------------------------------------------------
FATAL ERROR:
Both /bin/usb_printerid and /usr/bin/usb_printerid exist.

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.
---------------------------------------------------------------------------
and stuck with:
---------------------------------------------------------------------------------------------
sudo /usr/lib/usrmerge/convert-usrmerge
Smartmatch is experimental at /usr/lib/usrmerge/convert-usrmerge line 172.
 not a dynamic executable

FATAL ERROR:
Can't close(GLOB(0x55fc33b9c028)) filehandle: '' at /usr/lib/usrmerge/convert-usrmerge line 193

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.
---------------------------------------------------------------------------------------------

Running "sudo dpkg --configure -a", "sudo apt -f install" or "sudo /usr/lib/usrmerge/convert-usrmerge" give exactly the same error.

Later I have accidently install updates and now system refuses to boot:
----------------------------------------------------------------------------------------------------------
Kernel Panic : No working init found. Try passing init= option to kernel
----------------------------------------------------------------------------------------------------------