ERROR got an error from dpkg for pkg: 'libc6'

[Emanuele]

After the last upgrade I made this morning, at the error of dpkg on the libc6 package, I was no longer able to log in user: sudo apt update, full-upgrade or any login with "sudo", the message was "you have passed 3 attempts for wrong password ", at restart I tried to log in from the grub, recovery, I tried to click" repair package "with no result, not even to start a root shell was not successful, any attempt failed .
Luckily before the update I took a snapshot with Timeshift on Btrfs, so I have the snapshot still saved in case it was used to reproduce the bug, I also have logs saved, since I have separate subvolumes.
I enclose everything, if there are other details that can help you, I am available.
Thank you

log: https://paste.ubuntu.com/p/zSJcrYyW6H/

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: libc6 2.30-0ubuntu3
ProcVersionSignature: Ubuntu 5.4.0-14.17-generic 5.4.18
Uname: Linux 5.4.0-14-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu20
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Sat Mar 14 13:44:58 2020
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=it_IT.UTF-8
 SHELL=/bin/bash
SourcePackage: glibc
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

> 2020-03-14 13:10:05,654 DEBUG running apport_pkgfailure() libc6: 0.1332:il sottoprocesso installato pacchetto libc6:amd64 script post-installation ha restituito lo stato di errore 127

Can you please show the apt term log from this upgrade?

It is concerning that the log you posted so far does not show libcrypt1 being installed, despite this being a dependency of libc6. Did you already have libcrypt1 installed, and at what version?

[Andre Brait (andrebrait)]

I had libcrypt1 installed.

/lib/x86_64-linux-gnu/libcrypt.so existed, but /lib/x86_64-linux-gnu/libcrypt.so.1 didn't.

The solution in https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1866844 worked for me.

I'm not sure what you said on that bug about this not being the same thing experienced by people in the focal release pocket. AFAIK it's exactly the same. It's what just happened to me, but I will happily provide logs or anything in order to help fixing this.

Just let me know the commands.

[Jeffrey Bouter (jbouter)]

I was affected and posted the workaround in the other, above mentioned, bug. Here's my term.log - which also contains data of the - eventually - finished upgrade. Please note I upgraded today (2020-03-14), the last upgrade before that was two days ago (2020-03-12).

[RobertH (robert-hunt)]

Yes, I couldn't confirm any details about the bug yesterday (on https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1866844) coz I'm still locked out of my system but it was the only mention I could find then of a libc6/libcrypt install failure. Will Jeffrey's instructions there still work if using ZFS instead of ext4?

[Mark Shuttleworth (sabdfl)]

Added another term.log, hope that's useful

[Emanuele (emanuc)]

These are the apt logs: https://paste.ubuntu.com/p/BbqhVpYkwm/

I can play the update, I still have the Btrfs snapshot after the update that caused the problem, I restored the system state just before upgrading, so I can also try upgrading again if it is state and when it will be resolved.

[Kai Mast (kai-mast)]

Is there a way to get sudo working again without going through grub?

[Kai Mast (kai-mast)]

Sorry.. to clarify my last post. My system is still running. Should I go down the grub/livecd route or might be an easier way to do this?

[Mark Shuttleworth (sabdfl)]

@jbouter you are a lifesaver. Or at least, a weekend-saver, thank you :)

[Mark Shuttleworth (sabdfl)]

@kai-mast I think grub is the only way, since both sudo and login are stuck under these circumstances.

[Steve Langasek (vorlon)]

From Jeffrey's log:

Preparing to unpack .../libcrypt1_1%3a4.4.10-10ubuntu1_amd64.deb ...
Unpacking libcrypt1:amd64 (1:4.4.10-10ubuntu1) ...
Setting up libcrypt1:amd64 (1:4.4.10-10ubuntu1) ...

Yes, this shows that you had the buggy libcrypt1 installed, which satisfies the dependency of libc6 but has a problem that libc6 and libcrypt1 don't agree on the path of libcrypt.so.1 in the dpkg database, causing the Replaces: to go wrong and remove this file from disk.

Specifically, the old version of libcrypt1 considers /usr/lib/$arch/libcrypt.so.1 to be the authoritative path of this file. The new version of libcrypt1 has been fixed to match libc6, and consider /lib/$arch/libcrypt.so.1 as the authoritative path of the file. If your system has usrmerge, then /lib is a symlink to /usr/lib on the filesystem, so these paths resolve to the same thing, but dpkg doesn't know it, so cannot correctly handle the upgrade.

The wrong path in libcrypt1 was resolved in libxcrypt 1:4.4.10-10ubuntu4, which is the version that just migrated to the release pocket together with glibc 2.31.

Unfortunately, libxcrypt 1:4.4.10-10ubuntu1 somehow managed to make it into the release pocket earlier (it shouldn't have, and I don't know exactly how this happened), so some users already had it installed. And if they had this wrong version installed, libc6 does not force upgrade to the fixed version of libcrypt1 first, so you still hit the bug.

After you've hit this problem, installing/reinstalling the *new* version of libcrypt1 (1:4.4.10-10ubuntu4) should be enough to correct the failure.

It will take some time to sort out an archive-side solution that would prevent propagating this damage to more users of focal who had the buggy libcrypt1 version installed. At minimum it's going to require a reupload of glibc with a bumped versioned dependency on libcrypt1, and probably one of libxcrypt as well to manually handle the dpkg database breakage.

[Jeffrey Bouter (jbouter)]

Glad I was able to assist. Good to hear the fix is out there now. The (few, I hope) people affected will hopefully be redirected to the workaround posted in the other bug report.

@sabdfl Wish I was a lifesaver in times like these. But I'll take weekend saver ;-) Thank you for the compliment! :-)

[Kai Mast (kai-mast)]

Thanks everybody! I ended up taking the long route through grub. Two additions to the solution from the other bugreport:

1. My root disk is encrypted so I had to use the "ro init=/binb/bash" flag instead to get to the password promt. If you use `rw` you (or at least my setup) get stuck before bash starts.

2. You might need to fetch a bunch of packages over the network to actually run `apt -f install`. To get your Ethernet working execute `dhclient &` and set a nameserver in `/etc/resolv.conf`.

[Jeffrey Bouter (jbouter)]

Thanks for the addition Kai. I tested this on an unencrypted VM, but using `rw` in GRUB may not be the best option to begin with anyway, seeing as in my guide I remount / as `rw` anyway. `ro` might have been better indeed.

With regards to /etc/resolv.conf, keep in mind that this is usually a symlink to systemd-resolved, located in /run - booting to single-user mode through GRUB means systemd-resolved isn't running and most likely can't be started. It might be best to first remove /etc/resolv.conf (the dead symlink), and then create a new /etc/resolv.conf file with a nameserver in it.

[BertN45 (lammert-nijhof)]

I do weekly backup for my VMs, so I first update/upgrade all VMs before the backup. The first two systems passed without problems (Host OS Ubuntu 20.04 on ZFS and Xubuntu 20.04 VM on ext4). The next two failed a few minutes later (Ubuntu 20.04 VM and Ubuntu Mate 20.04 VM). I restored the zfs snapshot from last week and they work again with that old version, but retries fail again.
The first two systems have libcrypt1 1:4.4.10-10ubuntu4 and the others that failed have 1:4.4.10-10ubuntu1, despite that they have been updated only a few minutes later.

System that I did not touch on Saturday still have 1:4.4.10-10ubuntu1 and they think it is the latest version according to synaptic and the same is true for the systems I restored from a snapshot of the previous Saturday, March 7.

It looks that your older upgrade from early Saturday worked, but somewhere during the morning it did fall apart.

[BertN45 (lammert-nijhof)]

I tried apt, apt-get and synaptic, but there is no normal way to reinstall/install the new version of libcrypt1. Even synaptic force package version did not work, so I decided to wait for your correction.

[RobertH (robert-hunt)]

Second day unable to login. Didn't get any advice here about zfs (obviously not many users), but I'm in a recovery terminal and managed to find `zfs mount -a` which at least filled out \var. \lib is a symlink to \usr\lib but not finding a `x86_64-linux-gnu` folder there, so unable to follow Jefrrey's instructions about `libcrypt.so`. Any help would be appreciated coz I've run out of things to try.

[evils (evils)]

just my two cents:
rolling back zfs to a previous snapshot (automatic, 2020-03-14T21:39+1)
(via the grub menu (press `esc` once after the bios screen))(system only)
, i could log back in

after rolling back i tried to upgrade again in hopes of this being fixed

here's a link to some pictures i took of the screen, they're not in order and contain some stuff from recovery>dpkg from before the rollback, sry for not attaching them: https://imgur.com/a/dmgSbNj

before the rollback the most apparent issue was anything using `sudo` resulted in immediate 3 failed tries error, the same thing happened in the gui on the login screen
after the recreation on my second attempt to log in this error changed to `... account locked?`

and FWIW, /lib/x86_64-linux-gnu/libcrypt.so.1 is a link to libcrypt.so.1.1.0 in the same directory

[Jeffrey Bouter (jbouter)]

@RobertH, are you using a 64-bit architecture? if not, the folder may be named differently. Try running: ls -ld /lib/*-linux-gnu to see what the directory on your system may be called.

[RobertH (robert-hunt)]

Yeah, it's 64-bit. Just got in :-), thanks Jeffrey. Might have been finger problems earlier -- seems `x86_64-linux-gnu` was there, just not showing up in autocomplete. The `zfs mount -a` was the only extra step needed for ZFS. Thanks to everyone for your help. I've loved using Ubuntu for many years now.

[Emanuele (emanuc)]

I confirm that it has been resolved, I made the update without problems.
Thank you

[Rik Mills (rikmills)]

Fixed in:

glibc (2.31-0ubuntu6) focal; urgency=medium

  * Bump dependency on libcrypt1 to the version which fixes the path to
    libcrypt.so.1, to avoid files disappearing due to replaces on upgrade.
    LP: #18673431.

 -- Steve Langasek <email address hidden> Sat, 14 Mar 2020 16:21:20 -0700

[BertN45 (lammert-nijhof)]

Problem solved in all my Ubuntu Family members (Ubuntu, Ubuntu Mate,
Ubuntu Budgie, Kubuntu and Lubuntu)

On Sun, 2020-03-15 at 12:07 +0000, Emanuele wrote:
> I confirm that it has been resolved, I made the update without
> problems.
> Thank you
>

[John Feole (jfeole)]

This bug caused me to lose 3 desktops in rapid succession when they upgraded yesterday.

Like others,after finding this bug, and using comments 12 and 13, and the livecd boot, was able to recover all 3 of them..after hours of wasted time.

I am an Ubuntu user from the earliest revisions, and wonder, how was this released in this
state? As right when it installs, it disables the desktop, and when you reboot, login is broken.. Was it even tested??

[John Feole (jfeole)]

Note, my fix in pprior note was drived from: Bug: 1866844..

[RobertH (robert-hunt)]

Jsut realised that when I deleted the resolv.conf symlink and instead created a file containing the single line `nameserver 8.8.8.8` in order to get internet access from the broken system, that change remained once I got the system working again. But I forgot to note down what it was pointing to before. I see on my eoan system it's a symlink to `../run/systemd/resolve/stub-resolv.conf` so I presume that's also correct for focal? Just noting it here for other users to remember to fix afterwards (by deleting the file with `cd /etc/`. `sudo rm resolv.conf`, and then `sudo ln -s ../run/systemd/resolve/stub-resolv.conf resolv.conf' (although not sure why it's a relative path and not an absolute one). Searching Google for automatic ways to do this didn't lead me to any more automated way that would work on my focal system which doesn't appear to have the resolvconf package installed.

[cosine (mvanross)]

For those of us who have upgraded, but not yet rebooted,
- can we just wait for the next update and install that,
or
- somehow undo the update?

Thanks,

[Matthias Klose (doko)]

if you didn't yet reboot, re-install the libcrypt1 package:

apt install --reinstall libcrypt1

$ ls -l /lib/x86_64-linux-gnu/libcrypt.so.1*
lrwxrwxrwx 1 root root 17 Mar 10 17:24 /lib/x86_64-linux-gnu/libcrypt.so.1 -> libcrypt.so.1.1.0
-rw-r--r-- 1 root root 202760 Mar 10 17:24 /lib/x86_64-linux-gnu/libcrypt.so.1.1.0

if you didn't yet upgrade from an earlier focal version, make sure that you remove the libcrypt1 package before the upgrade. Then do a normal upgrade.

Upgrades from bionic or eoan are not affected.

[Richard Paul (two00lbwaster)]

> if you didn't yet upgrade from an earlier focal version, make sure that you remove the libcrypt1 package before the upgrade. Then do a normal upgrade.

You say that like it's a fact but I just tried to upgrade from disco to eoan to fossa and have this issue it seems.

Also, I can't fix this, my machine that has been killed by this is WSL2 and I have no ability to boot into single user mode that I'm aware of.

[Yves Lavoie (yves-lavoie-ing)]

eoan IS affected here too.

Now:
- libc6 refuses to complete installation because libcrypt doesn't provide XCRYPT_2.0
- libcrypt refuses to complete installation because libc6 is not yet configured.

Nice catch22

[Alex DeLorenzo (alex-delorenzo)]

This is a problem on Docker with armhf hosts when building new images using the `ubuntu:focal` image from Docker Hub[1]. Images will fail to build because of the reported error

I've included the error that results from running `docker build` on a Dockerfile referencing `ubuntu:focal` and installing libcrypt1

Images will build using the `ubuntu:rolling` image.

Host details:
Ubuntu 18.04.4 LTS
Docker version 19.03.11, build 42e35e6

[1] Exact image: https://hub.docker.com/layers/ubuntu/library/ubuntu/focal/images/sha256-5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8?context=explore

[Manfred Hampl (m-hampl)]

@Alex DeLorenzo: What you see is Bug #1867675 and has nothing to do with the issue of this bug report.

[Roberto Franceschini (franceschini-roberto)]

This seems to be still an issue, not in Ubutuntu, but maybe back in Debian? I have god the same problem on Raspbian so the problem may lie upstream in Debian ? I was updating to libc62.32 and libcrypt1:4.4.27 ... working on a fix right now using the trick to create the symlink for libcrypto.so.1