Upgrade fails due to "text file busy"

[David Kastrup]

I cannot upgrade to the current impish, getting the following error (even in a root shell in recovery mode):

$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libc6 linux-headers-lowlatency linux-lowlatency
The following packages will be upgraded:
  libc6 linux-headers-lowlatency linux-lowlatency
3 upgraded, 0 newly installed, 0 to remove and 1855 not upgraded.
8 not fully installed or removed.
Need to get 0 B/3032 kB of archives.
After this operation, 391 kB disk space will be freed.
Do you want to continue? [Y/n]
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)
Preconfiguring packages ...
(Reading database ... 683369 files and directories currently installed.)
Preparing to unpack .../libc6_2.34-0ubuntu2_amd64.deb ...
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):

Unpacking libc6:amd64 (2.34-0ubuntu2) over (2.33-0ubuntu5) ...
dpkg: error processing archive /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb (--unpack):
 unable to open '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new': Text file busy
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

There is no actual file '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new' left after the aborted install. Renaming the file '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2' manually turns out to be a rather bad idea. Not found a recipe to get past that so far.

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: libc6 2.33-0ubuntu5 [modified: usr/share/doc/libc6/NEWS.gz usr/share/doc/libc6/changelog.Debian.gz usr/share/lintian/overrides/libc6]
ProcVersionSignature: Ubuntu 5.13.0-14.14-lowlatency 5.13.1
Uname: Linux 5.13.0-14-lowlatency x86_64
ApportVersion: 2.20.11-0ubuntu65.1
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: XFCE
Date: Thu Sep 9 12:51:28 2021
InstallationDate: Installed on 2011-10-14 (3617 days ago)
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111011)
SourcePackage: glibc
UpgradeStatus: No upgrade log present (probably fresh install)

[Florian Weimer (fweimer)]

Is it possible that this is a result of running ldconfig early, before the `.dpkg-new` files have been renamed into place? If yes, can you stop doing that? If not, we could perhaps teach ldconfig to ignore .dpkg-new files. See _dl_is_dso in elf/dl-is_dso.h.

This may have been a result of the upstream removal of symbolic links. Before, those .dpkg-new files probably weren't created by dpkg.

[David Kastrup (dak)]

Some more info:

After the upgrade fails, there is no '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new' and there is no '/etc/ld.so.cache', meaning that I need to run 'ldconfig' to get back a mostly working system.

With no '/etc/ld.so.cache' after the failure (even if you started with one before calling 'apt'), I am not sure how likely it is that 'ldconfig' has been run too early.

[Florian Weimer (fweimer)]

The ldconfig aspect is just a guess on my part. Assuming that /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new was not created by renaming of /lib/x86_64-linux-gnu/ld-linux-x86-64.so, it is unclear to me how a process would end up using it—it has to be listed in /etc/ld.so.cache for that to happen.

[David Kastrup (dak)]

Florian Weimer <email address hidden> writes:

> The ldconfig aspect is just a guess on my part. Assuming that
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new was not created by
> renaming of /lib/x86_64-linux-gnu/ld-linux-x86-64.so, it is unclear to
> me how a process would end up using it—it has to be listed in
> /etc/ld.so.cache for that to happen.

Are you sure about that? My botched upgrade left me without
/etc/ld.so.cache in place and the system appeared mostly functional: the
main problem I saw was a refusal to execute a few self-compiled binaries
(relying on shared libraries in non-standard places) and probably a
number of GUI-related stuff (fonts were wrong size and so on). But
overall it was a "slightly defective but overall workable" situation
until I figure out I'd better run `ldconfig`.

So there must be a healthy dose of standard locations that are consulted
in the absence of the cache file.

--
David Kastrup

[Florian Weimer (fweimer)]

To me, the clue is the ETXTBSY for ld-linux-x86-64.so.2.dpkg-new (with the suffix). I can't quite see how this could become used without a reference from /etc/ld.so.cache. And I think according to the ldconfig heuristic, ld-linux-x86-64.so.2.dpkg-new is considered a newer version than ld-linux-x86-64.so.2. (Maybe we should not apply this heuristic if a file with the exact soname exists.)

[Michael Hudson-Doyle (mwhudson)]

Can I ask how you updated? I made a hirsute container and ran do-release-upgrade -d and didn't hit this, and I haven't seen any instances of this in e.g. autopkgtests or builds that upgrade from libc 2.33 to 2.34 within impish. But I can see how this might be some kind of race, and I guess it might be more likely if a larger set of packages are being updated. Could you attach your /var/log/apt?

[David Kastrup (dak)]

Here you are.

I actually got here via `do-release-update -d` (I tried adding a few packages manually via `dpkg` afterwards to get out of the current bad situation but they are likely not having much of an impact on the logs). If you say this may be a race condition, should I try reducing the system to a single core (it's a 4-core right now) via BIOS settings or whatever and try again? Problem of course being that if it works I cannot get back to the failing state again for further testing.

[David Kastrup (dak)]

Additional info: switched in the BIOS to single CPU with no hyperthreading and booted into recovery mode (and checked with cat /proc/cpuinfo that there was only a single CPU present to Linux). No change, still bombed out with the same error on apt --fix-broken install . So if there is a race condition, it would not appear to depend on having multiple CPUs even though of course there still is preemptive scheduling even with a single CPU.

[David Kastrup (dak)]

Ok, here is another possible problem source: I just did 'ls /etc/ld*' by accident and got this:

$ ls /etc/ld*
/etc/ld.so.conf /etc/ld.so.nohwcap

/etc/ldap:
ldap.conf

/etc/ld.so.conf.d:
fakeroot-i386-linux-gnu.conf i386-linux-gnu.conf libc.conf x86_64-linux-gnu.conf zz_amd64-biarch-compat.conf zz_i386-biarch-compat.conf.dpkg-new
fakeroot-x86_64-linux-gnu.conf i386-linux-gnu.conf.dpkg-new libc.conf.dpkg-new x86_64-linux-gnu_guile-1.8.conf zz_i386-biarch-compat.conf zz_x32-biarch-compat.conf

So there is a number of .dpkg-new files in /etc/ld.so.conf.d left over that might sabotage the update. Removing them now, rerunning ldconfig and trying again...

Nope. The .dpkg-new files remain gone but the upgrade still fails in the same manner and there is no /etc/ld.so.cache anymore afterwards.

[Michael Hudson-Doyle (mwhudson)]

Well this is very strange. Can you try running strace-ing the apt install -f invocation (something like "strace -ff -o apt.trace apt --fix-broken install" and sending me all the apt.trace.* files)?

[David Kastrup (dak)]

I did and sent the tar file encrypted to your Gmail account. Hope you got it.

Another thing: libc6-bin is already installed in the new version, and calling `locale` gives an error message like

    locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)

The `apt --fix-broken install` run complains about this. I removed the locale binary temporarily to get

Can't exec "locale": No such file or directory at /usr/share/perl5/Debconf/Encoding.pm line 16.
Use of uninitialized value $Debconf::Encoding::charmap in scalar chomp at /usr/share/perl5/Debconf/Encoding.pm line 17.

I have no idea what this is being used for but maybe it is in some way related to unexpected things happening and culminating in the listed error?

[Michael Hudson-Doyle (mwhudson)]

Hi, yes, I got the logs thanks. Haven't had time to more than glance at them so far, hopefully can spend some time with them tomorrow.

[David Kastrup (dak)]

Downgraded libc-bin to Focal, then did apt --fix-broken install . It attempted to install both libc-bin and libc6 and failed with the same error as previously while I didn't get the loader complaint associated with not being able to run /usr/bin/locale .

So the inability to execute /usr/bin/locale doesn't seem to be the principal problem but more a side effect of the broken state after libc-bin managed to get upgraded and libc6 didn't.

[Michael Hudson-Doyle (mwhudson)]

Well the strace output helps a bit but still doesn't really make sense. Can you try "apt install -o dpkg::options::=-D7200 -f" and attach the output? It will be quite large I think.

[David Kastrup (dak)]

Not really? It's just
apt install -o dpkg::options::=-D7200 -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libc-bin libc6
The following packages will be upgraded:
  libc-bin libc6
2 upgraded, 0 newly installed, 0 to remove and 1884 not upgraded.
6 not fully installed or removed.
Need to get 0 B/4050 kB of archives.
After this operation, 225 kB disk space will be freed.
Do you want to continue? [Y/n]
Preconfiguring packages ...
(Reading database ... 676266 files and directories currently installed.)
Preparing to unpack .../libc6_2.34-0ubuntu2_amd64.deb ...
D000200: process_archive conffile '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' in package libc6:amd64 - conff ?
D000200: pkg_conffiles_mark_old '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' namenode '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' flags 5
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):

Unpacking libc6:amd64 (2.34-0ubuntu2) over (2.33-0ubuntu5) ...
D000200: conffderef in='/etc/ld.so.conf.d/x86_64-linux-gnu.conf' current working='/etc/ld.so.conf.d/x86_64-linux-gnu.conf'
D000200: tarobject conffile extracted
dpkg: error processing archive /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb (--unpack):
 unable to open '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new': Text file busy
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[David Kastrup (dak)]

It was not an option to keep my system broken for longer, so I mounted my system disk to a working Ubuntu installation and used 'dpkg --root=/mnt --force-script-chrootless -i ...' from there for getting libc installed, after which a native apt install -f was able to take it from there.

Let's hope that this was really really a one-off error nobody else will see because I will not be able to provide any further information with something like 1500+ packages now having been upgraded.

[Michael Hudson-Doyle (mwhudson)]

Totally fair, and thanks for the information you have provided. Hopefully this was something more or less unique to your system...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sergey Korablin (s-korablin)]

I have the same problem after do-release-upgrade -d. Can provide some additional info if you need it

[Michael Hudson-Doyle (mwhudson)]

Oh huh. Can you run "apt install -f" with dpkg logging jacked all the way up? I thought that "apt install -o dpkg::options::=-D7200 -f" would do this but according to David this wasn't quite the right invocation. Also running apport-collect 1943124 should attach some basic information about your system to the bug which might be relevant.

[Sergey Korablin (s-korablin)]

apport-collect shows me an error that i'm not the reporter or subscriber of this problem report...

# apt install -o dpkg::options::=-D7200 -f
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libllvm11:i386 linux-headers-5.11.0-18 linux-headers-5.11.0-22 linux-headers-5.11.0-22-generic linux-headers-5.11.0-22-lowlatency linux-headers-5.11.0-25 linux-headers-5.11.0-25-generic linux-headers-5.11.0-25-lowlatency
  linux-image-5.11.0-22-generic linux-image-5.11.0-22-lowlatency linux-image-5.11.0-25-generic linux-image-5.11.0-25-lowlatency linux-modules-5.11.0-22-generic linux-modules-5.11.0-22-lowlatency linux-modules-5.11.0-25-generic
  linux-modules-5.11.0-25-lowlatency linux-modules-extra-5.11.0-22-generic linux-modules-extra-5.11.0-25-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libc6
Suggested packages:
  glibc-doc
The following packages will be upgraded:
  libc6
1 upgraded, 0 newly installed, 0 to remove and 1400 not upgraded.
7 not fully installed or removed.
Need to get 0 B/3 028 kB of archives.
After this operation, 391 kB disk space will be freed.
Do you want to continue? [Y/n] Y
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)
Preconfiguring packages ...
(Reading database ... 734128 files and directories currently installed.)
Preparing to unpack .../libc6_2.34-0ubuntu2_amd64.deb ...
D000200: process_archive conffile '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' in package libc6:amd64 - conff ?
D000200: pkg_conffiles_mark_old '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' namenode '/etc/ld.so.conf.d/x86_64-linux-gnu.conf' flags 5
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
/usr/bin/locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/locale)
Stopping some services possibly affected by the upgrade (will be restarted later):

Unpacking libc6:amd64 (2.34-0ubuntu2) over (2.33-0ubuntu5) ...
D000200: conffderef in='/etc/ld.so.conf.d/x86_64-linux-gnu.conf' current working='/etc/ld.so.conf.d/x86_64-linux-gnu.conf'
D000200: tarobject conffile extracted
dpkg: error processing archive /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb (--unpack):
 unable to open '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new': Text file busy
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.34-0ubuntu2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Michael Hudson-Doyle (mwhudson)]

Maybe try ubuntu-bug libc6 then, which will file a new bug -- but that's OK in this context I think.

[Yavor Nikolov (yavor-nikolov)]

I have the same problem while upgrading from Ubuntu 21.04 to 21.10

Outputs of:
strace -f -E --decode-fds=paths -o /tmp/dpkg-unpack-libc6.trace dpkg --unpack -D2000 --log=/tmp/dpkg-unpack-libc6.log /tmp/libc6_2.34-0ubuntu3_amd64.deb

[Simon Spannagel (simonspa)]

I also have the same issue here. Additional information I can offer is that I already has a weird situation when upgrading to 21.04, something of the usrmerge went wrong and I ended up not having a working ld. Not sure this is related, anyway.

Here's the output of running dpkg:

(Reading database ... 705532 files and directories currently installed.)
Preparing to unpack .../libc6_2.34-0ubuntu3_amd64.deb ...
locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by locale)
Checking for services that may need to be restarted...
Checking init scripts...
/usr/bin/locale: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/locale)
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):

Unpacking libc6:amd64 (2.34-0ubuntu3) over (2.33-0ubuntu5) ...
dpkg: error processing archive /var/cache/apt/archives/libc6_2.34-0ubuntu3_amd64.deb (--unpack):
 unable to open '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new': Text file busy
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.34-0ubuntu3_amd64.deb

[Yavor Nikolov (yavor-nikolov)]

I had a similar issue yesterday - not during release upgrade but when installing package updates (Ubuntu 21.10):

package libc6 2.34-0ubuntu3.2 failed to install/upgrade: unable to open '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.dpkg-new': Text file busy

The way I resolve it is via dual-boot, I boot to alternative installation, I mount the one which I want to update to let's say /mnt and I do:
dpkg -i --root=/mnt libc6.....deb

This leads to an error "too many levels of symbolic links" but after that I can login to my main installation and finish the package updates.

###
Update: I am sharing a bit more since I experienced more issues after the above steps.

Then I faced the same issue in my alternative environment: in attempt to fix it, I broke my main one... After a while I discovered that after attempts using chroot, dpkg -i --root=... - I get into an issue due to getting symlink like that:

/mnt/new/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

This leads to errors like "too many levels of symbolik links" on attempt to chroot, or kernel panik on attempt to bood directly from the affected environment.

To fix it, I do:
rm /mnt/new/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
cp -a /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /mnt/new/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

[Yavor Nikolov (yavor-nikolov)]

I think I found the culprit in my setup. The libc6 package (amd64) contains a symlink:
./lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

On my system I have a symlink to the whole folder:
/usr/lib64 -> /usr/lib/x86_64-linux-gnu

On another newer installation I can see that lib64 is a separate folder (not a symlink) with a single file inside it:
./lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

The newer installation has started from Ubuntu 20 or 21. And the old one - maybe from Ubuntu 5 which I have upgraded many times: I guess I somehow in the history of updates, I ended up with the above symlink (there might have been manual touches as well).

[Michael Hudson-Doyle (mwhudson)]

Ah yes that would do it. I wonder how that symlink got there -- I'm pretty sure it was not created by any official part of Ubuntu. (A bit of googling finds people doing things like this https://github.com/remap/ndnfs-port/issues/8 but it's not clear to me where they got the idea from).

We might be able to add a check for this in preinst or something I guess...

[Didier L (l-farquaad)]

Same issue for me on an old laptop installed from Ubuntu 14.04 according to /var/log/installer/media-info, which was itself installed on 2014-10-10… and /usr/lib64 was itself created (or updated) on 2014-10-15, so clearly not through the installer.

I already had previously some issues with usrmerge complaining about other symlinks, I guess it was somehow related:
 - /usr/lib64/ld-linux-x86-64.so.2
 - /bin/ksh

Note that fixing this does not seem trivial as (re)moving /usr/lib64 essentially breaks the system. I couldn’t use sudo any more (file not found for any command run as root), but I was expecting it so I had kept a root terminal open. From there, I was able to fix my issue by moving the new lib64 directory I had prepared:
/usr# lib64-new/ld-linux-x86-64.so.2 /usr/bin/mv lib64-new lib64

After recreating the /usr/lib64 directory, I am able to proceed with the upgrade (from 21.04 to 21.10 first).