Support upgrades from unmerged 22.04 - was: package libc6 2.35-0ubuntu3.6 [modified: usr/share/doc/libc6/NEWS.Debian.gz usr/share/doc/libc6/NEWS.gz usr/share/doc/libc6/README.hesiod.gz usr/share/doc/libc6/changelog.Debian.gz usr/share/lintian/overrides/libc6] failed to install/upgrade: new libc6:amd64 package pre-installation script subprocess returned error exit status 2

[Brian Wilson]

[Analysis]
usrmerge was not essential on 22.04 and systems that upgraded to it may not have been converted, as it was only Recommended.

We need to add quirks to ubuntu-release-upgrader to convert systems that are not yet merged by first installing the usrmerge package.

[Original bug report]
Doing upgrade from 22.04 -> 24.04 beta.

Upgrading
Fetched 0 B in 0s (0 B/s)
   Upgrading: libnih1:amd64 < 1.0.3-12build1 @ii mK Ib > due to libnih1:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (< 2.36)
   Upgrading: libc6-x32:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii ugH Ib > due to libc6-x32:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (= 2.35-0ubuntu3.6)
   Upgrading: libc6-i386:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii ugH Ib > due to libc6-i386:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (= 2.35-0ubuntu3.6)
   Upgrading: libc6-dev:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii umH Ib > due to libc6-dev:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (= 2.35-0ubuntu3.6)
   Upgrading: libc6-dbg:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii umH Ib > due to libc6-dbg:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (= 2.35-0ubuntu3.6)
   Upgrading: libc-dev-bin:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii umH NPb Ib > due to libc-dev-bin:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (< 2.36)
   Upgrading: libc-bin:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii umH Ib > due to libc-bin:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > (< 2.36)
  MarkInstall libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > FU=1
    MarkInstall locales:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > FU=0
    Installing libc-bin:amd64 as Depends of locales:amd64
      MarkInstall libc-bin:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU > FU=0
    MarkInstall libc6:i386 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU > FU=0
    MarkInstall libnih1:amd64 < 1.0.3-12build1 @ii mK Ib > FU=0
      libnih1:amd64 Depends on libc6:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii pumU > (< 2.36) can't be satisfied! (dep)
   Removing: libnih1:amd64 as upgrade is not possible
    MarkDelete libnih1:amd64 < 1.0.3-12build1 @ii mK Ib > FU=0
     Upgrading: libc6-dev-x32:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii ugH NPb IPb > due to libc6-dev-x32:amd64 Depends on libc6-x32:amd64 < 2.35-0ubuntu3.6 | 2.39-0ubuntu2 @ii ugH Ib > (= 2.35-0ubuntu3.6)
    MarkInstall libc6-x32:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii ugU > FU=0
      MarkInstall libc6-dev-x32:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii ugU NPb Ib > FU=0
      Installing libc6-dev-i386:amd64 as Depends of libc6-dev-x32:amd64
        MarkInstall libc6-dev-i386:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii ugU NPb Ib > FU=0
          MarkInstall libc6-dev:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU Ib > FU=0
          Installing libc-dev-bin:amd64 as Depends of libc6-dev:amd64
            MarkInstall libc-dev-bin:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU NPb IPb > FU=0
            ignore old unsatisfied important dependency on libc-devtools:amd64
        Installing libc6-i386:amd64 as Depends of libc6-dev-i386:amd64
          MarkInstall libc6-i386:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii ugU > FU=0
        ignore old unsatisfied important dependency on gcc-multilib:amd64
      ignore old unsatisfied important dependency on gcc-multilib:amd64
    MarkInstall libc6-dbg:amd64 < 2.35-0ubuntu3.6 -> 2.39-0ubuntu2 @ii umU > FU=0
Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) ureadahead:amd64 < 0.100.0-21 @ii mK Ib >
Broken ureadahead:amd64 Depends on libnih1:amd64 < 1.0.3-12build1 @ii mR > (>= 1.0.0)
  Considering libnih1:amd64 0 as a solution to ureadahead:amd64 -2
  Removing ureadahead:amd64 rather than change libnih1:amd64
  MarkDelete ureadahead:amd64 < 0.100.0-21 @ii mK Ib > FU=0
Done

Upgrading
Fetched 0 B in 0s (0 B/s)
Preconfiguring packages ...
Preconfiguring packages ...
Preconfiguring packages ...
(Reading database ... 361984 files and directories currently installed.)
Removing ureadahead (0.100.0-21) ...
Removing libnih1 (1.0.3-12build1) ...

Progress: [ 6%]
(Reading database ... 361965 files and directories currently installed.)
Preparing to unpack .../00-libc6-i386_2.39-0ubuntu2_amd64.deb ...
Unpacking libc6-i386 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...

Progress: [ 12%]
Preparing to unpack .../01-libc-dev-bin_2.39-0ubuntu2_amd64.deb ...
Unpacking libc-dev-bin (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../02-libc6-dev_2.39-0ubuntu2_amd64.deb ...

Progress: [ 18%]
Unpacking libc6-dev:amd64 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
Replacing files in old package libc6:amd64 (2.35-0ubuntu3.6) ...
Preparing to unpack .../03-libc6-dev-i386_2.39-0ubuntu2_amd64.deb ...
Unpacking libc6-dev-i386 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...

Progress: [ 24%]
Preparing to unpack .../04-libc6-dev-x32_2.39-0ubuntu2_amd64.deb ...
Unpacking libc6-dev-x32 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../05-libc6-x32_2.39-0ubuntu2_amd64.deb ...

Progress: [ 30%]
Unpacking libc6-x32 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../06-libc6-dbg_2.39-0ubuntu2_amd64.deb ...
Unpacking libc6-dbg:amd64 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...

Progress: [ 36%]
Preparing to unpack .../07-locales_2.39-0ubuntu2_all.deb ...
Unpacking locales (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../08-libc6_2.39-0ubuntu2_amd64.deb ...
De-configuring libc6:i386 (2.35-0ubuntu3.6), to allow configuration of libc6:amd64 (2.35-0ubuntu3.6) ...

Progress: [ 42%]
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):
  cron: stopping...done.

dpkg-divert: error: cannot divert directories

Use --help for help about diverting files.
dpkg: error processing archive /tmp/apt-dpkg-install-vdyTTj/08-libc6_2.39-0ubuntu2_amd64.deb (--unpack):
 new libc6:amd64 package pre-installation script subprocess returned error exit status 2
Preparing to unpack .../09-libc6_2.39-0ubuntu2_i386.deb ...
De-configuring libc6:amd64 (2.35-0ubuntu3.6), to allow configuration of libc6:i386 (2.35-0ubuntu3.6) ...

Progress: [ 48%]
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):

Unpacking libc6:i386 (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/libc6.0.crash'
                                                                               Preparing to unpack .../10-libc-bin_2.39-0ubuntu2_amd64.deb ...
Unpacking libc-bin (2.39-0ubuntu2) over (2.35-0ubuntu3.6) ...

Progress: [ 55%]
Errors were encountered while processing:
 /tmp/apt-dpkg-install-vdyTTj/08-libc6_2.39-0ubuntu2_amd64.deb
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (332.3 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C): S

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.....

*** Uploading problem information

The collected information is being sent to the bug tracking system.
This might take a few minutes.
99%

*** To continue, you must visit the following URL:

  https://bugs.launchpad.net/ubuntu/+source/glibc/+filebug/c8f0b42a-e780-11ee-a0e9-df24f41047cf?field.title=package+libc6+2.35-0ubuntu3.6+%5Bmodified%3A+usr%2Fshare%2Fdoc%2Flibc6%2FNEWS.Debian.gz+usr%2Fshare%2Fdoc%2Flibc6%2FNEWS.gz+usr%2Fshare%2Fdoc%2Flibc6%2FREADME.hesiod.gz+usr%2Fshare%2Fdoc%2Flibc6%2Fchangelog.Debian.gz+usr%2Fshare%2Flintian%2Foverrides%2Flibc6%5D+failed+to+install%2Fupgrade%3A+new+libc6%3Aamd64+package+pre-installation+script+subprocess+returned+error+exit+status+2

You can launch a browser now, or copy this URL into a browser on another computer.

ProblemType: Package
DistroRelease: Ubuntu 22.04
Package: libc6 2.35-0ubuntu3.6 [modified: usr/share/doc/libc6/NEWS.Debian.gz usr/share/doc/libc6/NEWS.gz usr/share/doc/libc6/README.hesiod.gz usr/share/doc/libc6/changelog.Debian.gz usr/share/lintian/overrides/libc6]
ProcVersionSignature: Ubuntu 5.15.0-101.111-generic 5.15.143
Uname: Linux 5.15.0-101-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Thu Mar 21 08:36:07 2024
ErrorMessage: new libc6:amd64 package pre-installation script subprocess returned error exit status 2
InstallationDate: Installed on 2018-05-26 (2125 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Python3Details: /usr/bin/python3.10, Python 3.10.12, python3-minimal, 3.10.6-1~22.04
PythonDetails: /usr/bin/python3.10, Python 3.10.12, python-is-python3, 3.9.2-2
RelatedPackageVersions:
 dpkg 1.21.1ubuntu2.3
 apt 2.4.11
SourcePackage: glibc
Title: package libc6 2.35-0ubuntu3.6 [modified: usr/share/doc/libc6/NEWS.Debian.gz usr/share/doc/libc6/NEWS.gz usr/share/doc/libc6/README.hesiod.gz usr/share/doc/libc6/changelog.Debian.gz usr/share/lintian/overrides/libc6] failed to install/upgrade: new libc6:amd64 package pre-installation script subprocess returned error exit status 2
UpgradeStatus: Upgraded to jammy on 2024-03-21 (0 days ago)

[Brian Wilson (bdwilson)]

Attempted removing diversions:

# dpkg-divert --quiet --remove --rename --package base-files --divert "/lib.usr-is-merged" "/lib"
# dpkg-divert --quiet --remove --rename --divert "/lib/ld-linux.so.2.usr-is-merged" "/lib/ld-linux.so.2"

# apt-get install -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libopts25 php8.2-mysql sntp
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libc6
Suggested packages:
  glibc-doc
The following packages will be upgraded:
  libc6
1 upgraded, 0 newly installed, 0 to remove and 2382 not upgraded.
10 not fully installed or removed.
Need to get 0 B/3,262 kB of archives.
After this operation, 172 kB disk space will be freed.
Do you want to continue? [Y/n] y
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by locale)
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by locale)
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by locale)
Preconfiguring packages ...
(Reading database ... 361971 files and directories currently installed.)
Preparing to unpack .../libc6_2.39-0ubuntu2_amd64.deb ...
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by locale)
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by locale)
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by locale)
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):

dpkg-divert: error: cannot divert directories

Use --help for help about diverting files.
dpkg: error processing archive /var/cache/apt/archives/libc6_2.39-0ubuntu2_amd64.deb (--unpack):
 new libc6:amd64 package pre-installation script subprocess returned error exit status 2
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.39-0ubuntu2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Julian Andres Klode (juliank)]

What does ls -lh / show?

[Julian Andres Klode (juliank)]

And ls -lh "/lib/ld-linux.so.2.usr-is-merged" "/lib/ld-linux.so.2"

[Julian Andres Klode (juliank)]

Please also include `dpkg -l` output, assuming you are fine with us knowing which packages are installed.

[Helmut Grohne (helmutg)]

I don't fully understand the situation at hand, but let me add some background.

From the log it seems fairly clear that `lib6.preinst` is the thing that fails here. It adds two diversions:

    dpkg-divert --quiet --add --no-rename --package base-files --divert "/.${rtlddir#/}.usr-is-merged" "$rtlddir"
    dpkg-divert --quiet --add --no-rename --divert "$rtlddir/$rtld_so.usr-is-merged" "$rtlddir/$rtld_so"

It seems fairly implausible that the "$rtlddir/$rtld_so" would become a directory for any reason. If that were the case, `dpkg-divert` (being a dynamic executable) would fail to start. Most likely, it is the first diversion that fails here. Since the package at hand is `:amd64`, `rtlddir=/lib64`. So for some reason, `/lib64` is a directory on your installation and not a symbolic link. In effect, this is (at least partially) an unmerged system and this is unsupported.

This still is a bug in the sense that the error message is very confusing. `libc6.preinst` should probably be verifying that `$rtlddir` really is a symbolic link before adding the diversion. In the end, the outcome would be the same: When `$rtlddir` is not a symbolic link, all that `libc6.preinst` can do is fail (with a better error message).

[Brian Wilson (bdwilson)]

# ls -lh /
total 2.1G
drwxr-xr-x 2 root root 4.0K Mar 18 17:26 bin
drwxr-xr-x 4 root root 4.0K Mar 19 16:34 boot
drwxrwxr-x 2 root root 4.0K May 26 2018 cdrom
drwxr-xr-x 2 root root 4.0K Feb 24 2021 code
-rw------- 1 root root 5.1M Jan 17 2020 core
drwxr-xr-x 20 root root 4.8K Mar 20 20:33 dev
drwxr-xr-x 189 root root 12K Mar 21 08:36 etc
drwxrwxrwx 3 root root 4.0K Feb 4 2023 evs_temp
drwxr-xr-x 15 root root 4.0K Feb 24 2021 home
lrwxrwxrwx 1 root root 34 Sep 30 2021 initrd.img -> boot/initrd.img-4.15.0-159-generic
lrwxrwxrwx 1 root root 34 Sep 30 2021 initrd.img.old -> boot/initrd.img-4.15.0-158-generic
drwxr-xr-x 19 root root 4.0K Mar 21 08:36 lib
drwxr-xr-x 2 root root 4.0K Jan 12 06:24 lib64
drwx------ 2 root root 16K May 26 2018 lost+found
drwxr-xr-x 4 root root 4.0K Dec 29 2022 media
drwxr-xr-x 9 root root 4.0K Jan 8 2023 mnt
drwxr-xr-x 3 root root 4.0K Jul 12 2020 new
drwxr-xr-x 16 root root 4.0K Jan 27 22:38 opt
dr-xr-xr-x 333 root root 0 Mar 20 20:33 proc
drwx------ 29 root root 4.0K Mar 21 09:05 root
drwxr-xr-x 51 root root 1.5K Mar 21 08:36 run
drwxr-xr-x 2 root root 4.0K Mar 21 08:36 sbin
drwxr-xr-x 20 root root 4.0K Dec 26 2022 snap
drwxr-xr-x 4 root root 4.0K Aug 11 2023 srv
drwxr-xr-x 4 wilson frontrow 4.0K Mar 18 17:02 storage
drwxr-xr-x 2 root root 4.0K May 27 2019 storage-new
drwxr-xr-x 5 root root 4.0K Jan 7 2023 storage.old
-rw------- 1 root root 2.0G May 26 2018 swapfile
dr-xr-xr-x 13 root root 0 Mar 20 20:33 sys
drwxrwxrwt 21 root root 4.0K Mar 21 14:09 tmp
drwxr-xr-x 15 root root 4.0K Dec 26 2022 usr
drwxr-xr-x 17 root root 4.0K Mar 13 12:57 var
lrwxrwxrwx 1 root root 31 Sep 30 2021 vmlinuz -> boot/vmlinuz-4.15.0-159-generic
lrwxrwxrwx 1 root root 31 Sep 30 2021 vmlinuz.old -> boot/vmlinuz-4.15.0-158-generic

# ls -lh "/lib/ld-linux.so.2.usr-is-merged" "/lib/ld-linux.so.2"
ls: cannot access '/lib/ld-linux.so.2.usr-is-merged': No such file or directory
ls: cannot access '/lib/ld-linux.so.2': No such file or directory

# ls -lat /lib
total 152
drwxr-xr-x 19 root root 4096 Mar 21 08:36 .
drwxr-xr-x 2 root root 4096 Mar 21 08:36 i386-linux-gnu
drwxr-xr-x 30 root root 4096 Mar 21 08:36 ..
drwxr-xr-x 51 root root 4096 Mar 19 16:33 modules
drwxr-xr-x 2 root root 4096 Mar 18 17:05 udev
drwxr-xr-x 90 root root 69632 Mar 18 17:04 firmware
drwxr-xr-x 2 root root 4096 Mar 18 17:04 ifupdown
drwxr-xr-x 3 root root 12288 Mar 18 17:04 x86_64-linux-gnu
drwxr-xr-x 4 root root 4096 Mar 18 17:03 systemd
drwxr-xr-x 2 root root 4096 Nov 19 13:13 netplan
drwxr-xr-x 2 root root 4096 Aug 18 2023 ufw
lrwxrwxrwx 1 root root 17 Dec 26 2022 apparmor -> /usr/lib/apparmor
lrwxrwxrwx 1 root root 19 Dec 26 2022 modprobe.d -> /usr/lib/modprobe.d
lrwxrwxrwx 1 root root 22 Dec 26 2022 recovery-mode -> /usr/lib/recovery-mode
lrwxrwxrwx 1 root root 16 Dec 26 2022 apcupsd -> /usr/lib/apcupsd
lrwxrwxrwx 1 root root 13 Dec 26 2022 in...

[Brian Wilson (bdwilson)]

dpkg -l output

[Brian Wilson (bdwilson)]

root@host:/lib/i386-linux-gnu# ls -lath
total 200K
drwxr-xr-x 2 root root 4.0K Mar 21 08:36 .
drwxr-xr-x 19 root root 4.0K Mar 21 08:36 ..
lrwxrwxrwx 1 root root 14 Jun 7 2023 libcap.so.2 -> libcap.so.2.44
-rw-r--r-- 1 root root 38K Jun 7 2023 libcap.so.2.44
-rw-r--r-- 1 root root 150K May 13 2023 libgcc_s.so.1
lrwxrwxrwx 1 root root 46 Dec 26 2022 libgpg-error.so.0.32.1 -> /usr/lib/i386-linux-gnu/libgpg-error.so.0.32.1
lrwxrwxrwx 1 root root 43 Dec 26 2022 libnss_nis.so.2.0.0 -> /usr/lib/i386-linux-gnu/libnss_nis.so.2.0.0
lrwxrwxrwx 1 root root 47 Dec 26 2022 libnss_nisplus.so.2.0.0 -> /usr/lib/i386-linux-gnu/libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root 40 Dec 26 2022 liblzma.so.5.2.5 -> /usr/lib/i386-linux-gnu/liblzma.so.5.2.5
lrwxrwxrwx 1 root root 44 Dec 26 2022 libdbus-1.so.3.19.13 -> /usr/lib/i386-linux-gnu/libdbus-1.so.3.19.13
lrwxrwxrwx 1 root root 41 Dec 26 2022 libcom_err.so.2.1 -> /usr/lib/i386-linux-gnu/libcom_err.so.2.1
lrwxrwxrwx 1 root root 41 Dec 26 2022 libcrypt.so.1.1.0 -> /usr/lib/i386-linux-gnu/libcrypt.so.1.1.0
lrwxrwxrwx 1 root root 41 Dec 26 2022 libtirpc.so.3.0.0 -> /usr/lib/i386-linux-gnu/libtirpc.so.3.0.0
lrwxrwxrwx 1 root root 42 Dec 26 2022 libkeyutils.so.1.9 -> /usr/lib/i386-linux-gnu/libkeyutils.so.1.9
lrwxrwxrwx 1 root root 20 Oct 25 2022 libdbus-1.so.3 -> libdbus-1.so.3.19.13
lrwxrwxrwx 1 root root 17 Jul 25 2022 libtirpc.so.3 -> libtirpc.so.3.0.0
lrwxrwxrwx 1 root root 17 Jun 1 2022 libcom_err.so.2 -> libcom_err.so.2.1
lrwxrwxrwx 1 root root 16 Apr 8 2022 liblzma.so.5 -> liblzma.so.5.2.5
lrwxrwxrwx 1 root root 23 Mar 24 2022 libnss_nisplus.so.2 -> libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root 19 Mar 24 2022 libnss_nis.so.2 -> libnss_nis.so.2.0.0
lrwxrwxrwx 1 root root 18 Feb 28 2022 libkeyutils.so.1 -> libkeyutils.so.1.9
lrwxrwxrwx 1 root root 22 Jan 18 2022 libgpg-error.so.0 -> libgpg-error.so.0.32.1
lrwxrwxrwx 1 root root 17 Dec 17 2021 libcrypt.so.1 -> libcrypt.so.1.1.0

[Brian Wilson (bdwilson)]

root@host:/lib/x86_64-linux-gnu# ls -lath ld*
-rwxr-xr-x 1 root root 236K Jan 2 08:22 ld-linux-x86-64.so.2

[Helmut Grohne (helmutg)]

Thank you. Quite clearly, that system is not /usr-merged, but should be. Not being /usr-merged is unsupported at this time. The question arises how you got there. Can you explain what kind of system that is? Is it a regular installation, a container, a chroot or something else? Was it created recently or upgraded since a long time? Do you happen to know whether debootstrap, cdebootstrap, mmdebstrap or something else was used to construct it?

[Brian Wilson (bdwilson)]

This system was a rsync clone of a baremetal 22.04 system with the intention of being the drive I test the 24.04 upgrade on before doing it, so I still have the source system on 22.04 if there is something you want me to investigate on the source. I suppose the rsync command may not be syncing correctly or I'm using the wrong flags.

The rsync was done

    rsync -qaxH --delete --ignore-errors / /mnt/backup
    rsync -qaxH --delete --ignore-errors /boot/efi /mnt/backup/boot

    sed -i "s/$SourceRootUUID/$TargetRootUUID/g" /mnt/backup/etc/fstab
    sed -i "s/$SourceRootUUID/$TargetRootUUID/g" /mnt/backup/boot/grub/grub.cfg
    sed -i "s/$SourceEFIUUID/$TargetEFIUUID/g" /mnt/backup/etc/fstab
    sed -i "s/$SourceEFIUUID/$TargetEFIUUID/g" /mnt/backup/boot/grub/grub.cfg
    sed -i "s/$SourceRootUUID/$TargetRootUUID/g" /mnt/backup/boot/efi/EFI/ubuntu/grub.cfg
    chroot /mnt/backup grub-install /dev/$DEV
    chroot /mnt/backup /usr/sbin/update-grub

The clone is booted on another baremetal intel 700p.

I've done nothing to specifically remove /usr-merged.

[Helmut Grohne (helmutg)]

Thank you. Can you please verify which of /bin, /lib, /lib64, /sbin are symlinks on that source system?

If they are all symlinks there, your rsync copy somehow managed to unmerge the system. If they are all directories there, please try to go into more detail how that baremetal 22.04 system was originally installed (if possible).

In order to fix up your situation, you should attempt to install the `usrmerge` package (possibly from an older release if it is no longer available from noble). Failing that, copy the script `/usr/lib/usrmerge/convert-usrmerge` from any other system and run it locally. You may need to install some Perl modules to have it work. After having performed this step, `libc6.preinst` should no longer fail and your upgrade should proceed normally.

Even after fixing this, I'm still interested in how you got into that situation and identifying the root cause.

[Julian Andres Klode (juliank)]

It seems the usrmerge package had not been installed on the 22.04 system in the first place, so not sure how that happened but it seems prudent to quirk it in the ubuntu-release-upgrader to convert first before upgrading.

[Helmut Grohne (helmutg)]

https://salsa.debian.org/helmutg/bootstrap-usrmerge-demo/-/commit/49a35aa03472277b80406e152ee96989d8081d45 is my change for the Debian side (which has not reached unstable yet) and adds a safe-guard to libc6.preinst. I think that would be reasonable to include in Ubuntu too.

[Brian Wilson (bdwilson)]

Thank you for your help triaging this! This is from the source system. It is possible that I upgraded from another LTS to 22.04. I've only used LTS releases. See below for usrmerge installation output on the source system.

$ ls -lath
total 2.1G
drwxrwxrwt 38 root root 36K Mar 22 09:04 tmp
drwxr-xr-x 55 root root 1.6K Mar 22 09:03 run
drwxr-xr-x 4 root root 4.0K Mar 20 06:33 boot
drwxr-xr-x 189 root root 12K Mar 20 06:32 etc
drwxr-xr-x 2 root root 4.0K Mar 20 06:32 bin
drwx------ 29 root root 4.0K Mar 19 19:39 root
drwxr-xr-x 21 root root 5.0K Mar 18 16:54 dev
drwxrwxrwx 17 wilson frontrow 4.0K Mar 16 07:41 storage
dr-xr-xr-x 534 root root 0 Mar 16 07:41 proc
dr-xr-xr-x 13 root root 0 Mar 16 07:41 sys
drwxr-xr-x 2 root root 4.0K Mar 15 15:16 sbin
drwxr-xr-x 17 root root 4.0K Mar 13 12:57 var
drwxr-xr-x 16 root root 4.0K Jan 27 22:38 opt
drwxr-xr-x 19 root root 4.0K Jan 12 06:24 lib
drwxr-xr-x 2 root root 4.0K Jan 12 06:24 lib64
drwxr-xr-x 2 root root 4.0K Jan 12 06:24 libx32
drwxr-xr-x 2 root root 4.0K Jan 12 06:24 lib32
drwxr-xr-x 4 root root 4.0K Aug 11 2023 srv
drwxr-xr-x 32 root root 4.0K Feb 4 2023 .
drwxr-xr-x 32 root root 4.0K Feb 4 2023 ..
drwxrwxrwx 3 root root 4.0K Feb 4 2023 evs_temp
drwxr-xr-x 9 root root 4.0K Jan 8 2023 mnt
drwxr-xr-x 5 root root 4.0K Jan 7 2023 storage.old
drwxr-xr-x 4 root root 4.0K Dec 29 2022 media
drwxr-xr-x 15 root root 4.0K Dec 26 2022 usr
drwxr-xr-x 20 root root 4.0K Dec 26 2022 snap
lrwxrwxrwx 1 root root 34 Sep 30 2021 initrd.img -> boot/initrd.img-4.15.0-159-generic
lrwxrwxrwx 1 root root 34 Sep 30 2021 initrd.img.old -> boot/initrd.img-4.15.0-158-generic
lrwxrwxrwx 1 root root 31 Sep 30 2021 vmlinuz -> boot/vmlinuz-4.15.0-159-generic
lrwxrwxrwx 1 root root 31 Sep 30 2021 vmlinuz.old -> boot/vmlinuz-4.15.0-158-generic
drwxr-xr-x 15 root root 4.0K Feb 24 2021 home
drwxr-xr-x 2 root root 4.0K Feb 24 2021 code
drwxr-xr-x 3 root root 4.0K Jul 12 2020 new
-rw------- 1 root root 5.1M Jan 17 2020 core
drwxr-xr-x 2 root root 4.0K May 27 2019 storage-new
drwxrwxr-x 2 root root 4.0K May 26 2018 cdrom
-rw------- 1 root root 2.0G May 26 2018 swapfile
drwx------ 2 root root 16K May 26 2018 lost+found

On the source system:
# apt-get install usrmerge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libopts25 sntp
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  usrmerge
0 upgraded, 1 newly installed, 0 to remove and 138 not upgraded.
Need to get 54.7 kB of archives.
After this operation, 205 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 usrmerge all 25ubuntu2 [54.7 kB]
Fetched 54.7 kB in 0s (314 kB/s)
Selecting previously unselected package usrmerge.
(Reading database ... 361142...