libc upgrade breaks everything

[Guillermo Pérez]

The upgrade of the libc6 package in hardy has segfaulted and now the system is unusable. A lot of programs seg-fault, including apt/dpkg!!!

The dpkg-log is:
2008-03-13 13:33:50 upgrade libc6-amd64 2.7-5ubuntu2 2.7-9ubuntu1
2008-03-13 13:33:50 status half-configured libc6-amd64 2.7-5ubuntu2
2008-03-13 13:33:50 status unpacked libc6-amd64 2.7-5ubuntu2
2008-03-13 13:33:50 status half-installed libc6-amd64 2.7-5ubuntu2
2008-03-13 13:33:52 status half-installed libc6-amd64 2.7-5ubuntu2
2008-03-13 13:33:52 status unpacked libc6-amd64 2.7-9ubuntu1
2008-03-13 13:33:54 status unpacked libc6-amd64 2.7-9ubuntu1
2008-03-13 13:33:54 upgrade libc6 2.7-5ubuntu2 2.7-9ubuntu1
2008-03-13 13:33:54 status half-configured libc6 2.7-5ubuntu2
2008-03-13 13:33:54 status unpacked libc6 2.7-5ubuntu2
2008-03-13 13:33:54 status half-installed libc6 2.7-5ubuntu2
2008-03-13 13:33:58 status half-installed libc6 2.7-5ubuntu2
2008-03-13 13:33:58 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:00 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:00 startup packages configure
2008-03-13 13:34:00 configure libc6 2.7-9ubuntu1 2.7-9ubuntu1
2008-03-13 13:34:00 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:00 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:00 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:00 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:01 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:01 status unpacked libc6 2.7-9ubuntu1
2008-03-13 13:34:01 status half-configured libc6 2.7-9ubuntu1
2008-03-13 13:34:09 startup packages configure
2008-03-13 13:34:09 configure libc6 2.7-9ubuntu1 2.7-9ubuntu1
2008-03-13 13:34:09 status half-configured libc6 2.7-9ubuntu1

If I try to execute dpkg to reinstall the libc package:
$ sudo dpkg -i /var/cache/apt/archives/libc6_2.7-9ubuntu1_i386.deb
*** glibc detected *** sudo: free(): invalid pointer: 0xb7ca6b28 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7e48a87]
/lib/libc.so.6(cfree+0x89)[0xb7e4a749]
/lib/libc.so.6[0xb7eda740]
/lib/libc.so.6(__libc_dlsym+0x36)[0xb7eda7c6]
/lib/libc.so.6(__nss_lookup_function+0x154)[0xb7eb5704]
/lib/libc.so.6[0xb7e8ef69]
/lib/libc.so.6(getaddrinfo+0x1a0)[0xb7e905d0]
sudo[0x804ef45]
sudo[0x8050000]
/lib/libc.so.6(__libc_start_main+0xe0)[0xb7df73a0]
sudo[0x804a521]
======= Memory map: ========
08048000-08062000 r-xp 00000000 08:05 4194533 /usr/bin/sudo
08062000-08063000 rw-p 00019000 08:05 4194533 /usr/bin/sudo
08063000-08087000 rw-p 08063000 00:00 0 [heap]
b7c6f000-b7c79000 r-xp 00000000 08:05 46137623 /lib/libgcc_s.so.1
b7c79000-b7c7a000 rw-p 0000a000 08:05 46137623 /lib/libgcc_s.so.1
b7c7a000-b7c83000 r-xp 00000000 08:05 46166457 /lib/libnss_files-2.7.so
b7c83000-b7c85000 rw-p 00008000 08:05 46166457 /lib/libnss_files-2.7.so
b7c85000-b7c8d000 r-xp 00000000 08:05 46166459 /lib/libnss_nis-2.7.so
b7c8d000-b7c8f000 rw-p 00007000 08:05 46166459 /lib/libnss_nis-2.7.so
b7c8f000-b7ca2000 r-xp 00000000 08:05 46166454 /lib/libnsl-2.7.so
b7ca2000-b7ca4000 rw-p 00012000 08:05 46166454 /lib/libnsl-2.7.so
b7ca4000-b7ca7000 rw-p b7ca4000 00:00 0
b7ca7000-b7cad000 r-xp 00000000 08:05 46166455 /lib/libnss_compat-2.7.so
b7cad000-b7caf000 rw-p 00005000 08:05 46166455 /lib/libnss_compat-2.7.so
b7cbe000-b7cfd000 r--p 00000000 08:05 50332038 /usr/lib/locale/en_US.utf8/LC_CTYPE
b7cfd000-b7cfe000 r--p 00000000 08:05 54656537 /usr/lib/locale/en_US.utf8/LC_NUMERIC
b7cfe000-b7cff000 r--p 00000000 08:05 29411206 /usr/lib/locale/en_US.utf8/LC_TIME
b7cff000-b7de0000 r--p 00000000 08:05 50432672 /usr/lib/locale/en_US.utf8/LC_COLLATE
b7de0000-b7de1000 rw-p b7de0000 00:00 0
b7de1000-b7f14000 r-xp 00000000 08:05 46166448 /lib/libc-2.7.so
b7f14000-b7f15000 r--p 00133000 08:05 46166448 /lib/libc-2.7.so
b7f15000-b7f17000 rw-p 00134000 08:05 46166448 /lib/libc-2.7.so
b7f17000-b7f1b000 rw-p b7f17000 00:00 0
b7f1b000-b7f1d000 r-xp 00000000 08:05 46166451 /lib/libdl-2.7.so
b7f1d000-b7f1f000 rw-p 00001000 08:05 46166451 /lib/libdl-2.7.so
b7f1f000-b7f28000 r-xp 00000000 08:05 46214124 /lib/libpam.so.0.81.6
b7f28000-b7f29000 rw-p 00008000 08:05 46214124 /lib/libpam.so.0.81.6
b7f29000-b7f2a000 r--p 00000000 08:05 29411207 /usr/lib/locale/en_US.utf8/LC_MONETARY
b7f2a000-b7f2b000 r--p 00000000 08:05 17335083 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
b7f2b000-b7f2c000 r--p 00000000 08:05 58779672 /usr/lib/locale/en_US.utf8/LC_PAPER
b7f2c000-b7f2d000 r--p 00000000 08:05 34710 /usr/lib/locale/en_US.utf8/LC_NAME
b7f2d000-b7f2e000 r--p 00000000 08:05 29411208 /usr/lib/locale/en_US.utf8/LC_ADDRESS
b7f2e000-b7f2f000 r--p 00000000 08:05 29411209 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
b7f2f000-b7f30000 r--p 00000000 08:05 33666313 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
b7f30000-b7f37000 r--s 00000000 08:05 46179909 /usr/lib/gconv/gconv-modules.cache
b7f37000-b7f38000 r--p 00000000 08:05 29411210 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
b7f38000-b7f39000 rw-p b7f38000 00:00 0
b7f39000-b7f3a000 r-xp b7f39000 00:00 0 [vdso]
b7f3a000-b7f54000 r-xp 00000000 08:05 46166445 /lib/ld-2.7.so
b7f54000-b7f55000 r--p 00019000 08:05 46166445 /lib/ld-2.7.so
b7f55000-b7f56000 rw-p 0001a000 08:05 46166445 /lib/ld-2.7.so
bf984000-bf999000 rw-p bffeb000 00:00 0 [stack]
Aborted

Any clue on how to recover the system? Running programs are ok, but new started programs always crash.

This bug could cause lot of problems to everybody.

[Luca Carrogu (motoplux)]

I can confirm this!
But my system cannot start, neither in recovery mode.

[Jeroen van Disseldorp (dizzl)]

I can confirm this. After a dist-upgrade this morning I also got the libc errors. I tried repairing but nothing worked. No new shells can be opened. Switching to terminal (ctrl-alt-f1) and trying to log in does not work. It just comes back with another "login name:" prompt.

Also, reboots fail: the system just does nothing anymore.

I just reinstalled Alpha 5 and will be monitoring libc for any bugfixes before upgrading again.

[André Lemos (alemos)]

how could this have gone through? :\

[Hannu Teulahti (teu)]

Boot from live cd. mount root filesystem somewhere. find and extract old version of libc6 package contents to root filesystem.

dpkg -x libc6_2.7-5ubuntu2_amd64.deb /where/you/mounted/the/root/fs/

this isn't perfect, but fixes the problem.

Remember that apt thinks the broken package is installed, but the files are actually from the older package.

If you in luck the old package is still in /var/cache/apt/archives/ of your rootfs.

[Guillermo Pérez (bisho)]

This is known and spoken on IRC and the forums: Check http://ubuntuforums.org/showthread.php?t=722886