package libcrypt1 1:4.4.10-10ubuntu4 failed to install/upgrade: rm command for cleanup subprocess returned error exit status 127

[Gabriel Tzagkarakis]

do-release-upgrade from bionic to focal has broken for me on at least 9 servers during release upgrade. perl tries to load /lib/x86_64-linux-gnu/libcrypt.so.1 ,which is deleted at this point,
and the upgrade fails.
 I always manage to complete the upgrade by copying /lib/x86_64-linux-gnu/libcrypt.so.1.1.0
from a healthy system (20.04.1) and making the symlink to /lib/x86_64-linux-gnu/libcrypt.so.1 and resuming the upgrade with apt dist-upgrade.

release before upgrade:
Description: Ubuntu 18.04.5 LTS
Release: 18.04

ProblemType: Package
DistroRelease: Ubuntu 20.04
Package: libcrypt1 1:4.4.10-10ubuntu4
ProcVersionSignature: Ubuntu 5.4.0-53.59-generic 5.4.65
Uname: Linux 5.4.0-53-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.11
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Nov 11 09:49:28 2020
DuplicateSignature:
 package:libcrypt1:1:4.4.10-10ubuntu4
 Unpacking libc6:amd64 (2.31-0ubuntu9.1) over (2.27-3ubuntu1.3) ...
 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
 dpkg: error while cleaning up:
  rm command for cleanup subprocess returned error exit status 127
ErrorMessage: rm command for cleanup subprocess returned error exit status 127
InstallationDate: Installed on 2015-11-12 (1825 days ago)
InstallationMedia: Ubuntu-Server 15.10 "Wily Werewolf" - Release amd64 (20151021)
Python3Details: /usr/bin/python3.8, Python 3.8.5, python3-minimal, 3.8.2-0ubuntu2
PythonDetails: /usr/bin/python2.7, Python 2.7.18, python-minimal, 2.7.15~rc1-1
RelatedPackageVersions:
 dpkg 1.19.7ubuntu3
 apt 2.0.2ubuntu0.1
SourcePackage: libxcrypt
Title: package libcrypt1 1:4.4.10-10ubuntu4 failed to install/upgrade: rm command for cleanup subprocess returned error exit status 127
UpgradeStatus: Upgraded to focal on 2020-11-11 (0 days ago)
modified.conffile..etc.default.apport: [modified]
mtime.conffile..etc.default.apport: 2015-11-12T15:12:24.016495

[Alan Orth (aorth)]

I just hit this issue on a server as well. I had to copy the libcrypt.so.1.1.0 from a healthy Ubuntu 20.04 server via HTTP, as scp wasn't working so I assumed anything using https wouldn't either:

# wget http://server:8000/libcrypt.so.1.1.0
# mv libcrypt.so.1.1.0 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
# ln -s /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0 /usr/lib/x86_64-linux-gnu/libcrypt.so.1
# apt install -f
# apt dist-upgrade
# reboot

After that the server was OK.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Gillespie (markg1234)]

Year old bug that leaves you with a borked upgrade and no official fix?

[Mark Gillespie (markg1234)]

Method from Alan Orth (aorth) worked for me. To clarify, you need to get a copy of libcrypt.so.1.1.0 BEFORE attempting upgrade, as by the time it borks, you might have trouble getting the file onto the system, as many utils won't run.

For me, I got the file from here and extracted it:

http://mirrors.edge.kernel.org/ubuntu/pool/main/libx/libxcrypt/libcrypt1_4.4.10-10ubuntu4_amd64.deb

When the upgrade failed, cancelled out (x), and then copy, symlink and resume upgrade as detailed above.

[Gabriel Tzagkarakis (gabrieltz)]

This issue occured again today when upgrading another server from 18.04 to 20.04

here is a snippet and i'll try to guess what may be happening

==========
Stopping some services possibly affected by the upgrade (will be restarted later):
  cron: stopping...done.

Unpacking libc6:amd64 (2.31-0ubuntu9.9) over (2.27-3ubuntu1.5) ...

Progress: [ 16%]
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error while cleaning up:
 rm command for cleanup subprocess returned error exit status 127
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error processing archive /tmp/apt-dpkg-install-HpZRay/04-libcrypt1_1%3a4.4.10-10ubuntu4_i386.deb (--unpack):
 rm command for cleanup subprocess returned error exit status 127
=======

We had i386 architecture installed on this server due to an old 32bit program, could this be caused by the upgrade of the 32bit version of libcrypt1 ?

We also had a systemd unit running swatch on some log files which runs with perl which was not stopped like cron did , could this also cause perl to fail ?

My last guess would be if that safe-rm is installed somehow doesn't return correctly and should be removed before the upgrade

I hope my observations helps you solve this issue

[Gabriel Tzagkarakis (gabrieltz)]

I was able to reproduce this issue. The problem occurs when safe-rm is installed.

How to reproduce using lxc:

lxc launch images:ubuntu/18.04 bughunt
lxc exec bughunt bash

once you're inside the container run:

apt install ubuntu-release-upgrader-core safe-rm
do-release-upgrade

answer yes so that the upgrade initiates

when the upgrader reaches "Unpacking libc6:amd64 (2.31-0ubuntu9.9)" it fails
with
===============
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error while cleaning up:
 rm command for cleanup subprocess returned error exit status 127
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error processing archive /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_amd64.deb (--unpack):
 rm command for cleanup subprocess returned error exit status 127
Errors were encountered while processing:
 /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_amd64.deb
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)
=========

it appears that a quick workaround is to uninstall safe-rm before upgrading and reinstall afterwards

thank you