REGRESSION: glibc 2.7-9ubuntu1 NSS module broken due to toolchain changes

Bug #201673 reported by TheGZeus on 2008-03-13
This bug affects 2 people
Affects Status Importance Assigned to Milestone
glibc (Ubuntu)

Bug Description

This bug affects the Hardy development release (to become Ubuntu 8.04 LTS) ONLY. Stable releases of Ubuntu are NOT affected.

In glibc 2.7-9ubuntu1, many critical programs (such as bash and sudo) fail to run with various error messages, such as "malloc: ../bash/subst.c:3472: assertion botched" from bash and "*** glibc detected *** sudo: free(): invalid pointer: 0xb7fabb70 ***" from sudo. This renders the system unusable. This bug was introduced due to changes in the default value of LDFLAGS for package builds; while these changes were useful and had been tested elsewhere, they unexpectedly broke the C library when it was next built.

The quickest workaround is to copy an older version of the C library from the initramfs. This will work provided you upgraded from a relatively recent version of Hardy, so that the initramfs really does have an older version. DO NOT do this if you encountered this bug immediately after upgrading from Ubuntu 7.10 or earlier, as it will break your system badly. Reboot if necessary, and follow these steps:

 * Press Escape at start-up time to access the GRUB menu (this is not necessary on all systems)
 * Press 'e' to edit the normal Ubuntu boot options
 * Use the cursor keys to reach the line starting with 'kernel'
 * Press 'e' again to edit the boot command line
 * Change 'ro' to 'rw', remove 'splash', and add 'break=bottom' (without the quotes) to the end of the boot options
 * Press Enter and then 'b' to start up
 * After a few moments, you will be presented with an '(initramfs)' prompt
 * Type the following (you may see some 'No such file or directory' messages after running the cp command, which you can safely ignore):
     mount -o remount,rw /root
     cp /lib/ /lib/ /lib/ /lib/ /lib/ /root/lib/
     # run the following on the 32-bit PC edition (i386)
     cp /lib/ /root/lib/
     # run the following on the 64-bit PC edition (amd64)
     cp /lib64/ /root/lib64/
     umount /root
 * Start up normally

If this does not work, or if you encountered this bug immediately after upgrading from Ubuntu 7.10 or earlier, then you will need to follow the steps below instead.

If you have turned off your system, then you will be unable to start up and log in normally. In that case, get hold of a Hardy Alpha 6 desktop CD (since you are running a development version of Ubuntu, this should be reasonable); you can download it from if you do not have one to hand, and follow these steps:

 * Start up from the desktop CD
 * Run Applications -> Accessories -> Terminal (GNOME) or K-Menu -> System -> Konsole (KDE)

 * If you are running the 32-bit PC (i386) edition of Ubuntu, type the following into the terminal window:
 * If you are running the 64-bit PC (amd64) edition of Ubuntu, type the following into the terminal window:
 * If you are running some other edition of Ubuntu, older packages may be available from

 * In any case, continue by finding out the device name for your root partition, which will look something like '/dev/sda1'. You can run 'sudo fdisk -l' to find out all the available device names. The one you want will have 'Linux' in the 'System' column. A default installation will only have one such device; if you have more than one, then you probably know what you are doing!
 * Type the following into the terminal window, replacing DEVICE with the device name you found above:
     sudo mount DEVICE /mnt
     sudo mount --bind /dev /mnt/dev
     sudo mount --bind /proc /mnt/proc
     sudo cp libc6_*.deb /mnt/tmp/
     # replace 'i386' with 'amd64' if you are running the 64-bit PC edition
     sudo chroot /mnt dpkg -i /tmp/libc6_2.7-5ubuntu2_i386.deb
     # this step may return some errors; don't worry about them
     sudo chroot /mnt dpkg --configure -a
     sudo chroot /mnt update-initramfs -u
     sudo umount /mnt/proc
     sudo umount /mnt/dev
     sudo umount /mnt
 * Reboot

Fixed packages (version 2.7-9ubuntu2) are now on However, many mirrors will not yet have picked this up, so we advise that you be careful about upgrades for the next couple of days to avoid reintroducing the bug.

Changed in glibc:
importance: Undecided → Critical
status: New → Confirmed
Jeffrey Baker (jwbaker) wrote :

Confirmed. This is EXTREME bad juju. Nothing works. I, too, am afraid to shut down. This is libc6 2.7-9ubuntu1 on Hardy i386. The system went into the crapper during an apt-get update. The previous version of libc was 2.7-5ubuntu2.

Kyle M Weller (kylew) wrote :

i confirm this bug, had to boot into live cd and dpkg -x oldlibc /pathtomounted borked system to boot back up

TheGZeus (gzeusmants) wrote :

any more details on the exact procedure of that workaround? That's hilarious, but I'm cowering.

Vikrant (vikrant82) wrote :

This was really scary. On a reboot I couldn't even get a shell in recovery mode.

"malloc: unknown:0: assertion botched
free: called with unallocated block argument
last command: sudo mount
Aborting... init: tty1 main process ended, respawning"

Anyways, people have talked about solutions here.

Steve Langasek (vorlon) on 2008-03-13
Changed in glibc:
milestone: none → ubuntu-8.04-beta
Mario Limonciello (superm1) wrote :

Oh noes. I'm hit by this too. I don't want to reboot...

Jeffrey Baker (jwbaker) wrote :

I was able to recover my system without a reboot by installing the packages, still in the archive, of version 2.7-5ubuntu2. For whatever reason, su is able to run while sudo is not. Obviously this will only help you if you have a root password.

Enver ALTIN (ealtin) wrote :

If you don't have a root password, a solution that I used to downgrade the package which involves a reboot follows:

- If you have upgraded, but didn't reboot yet, get the older version from your favorite mirror and save it to somewhere you know. An example link is
- If you can't launch a browser for whatever reason, use alt+f2, type this: uxterm -e /bin/dash and use links to download.
- reboot.
- on the grub, select your favorite kernel, hit the 'e' key, select the line starting with 'kernel', change the 'ro' word on the line to 'rw' and add 'init=/bin/dash' to the end.
- wait for 10 seconds, nothing will be displayed when the system is booted into a root shell. try 'ls' to see whether it's done.
- cd /to/the/path/you/downloaded/the/package and use dpkg to downgrade: 'dpkg -i libc6_2.7-5ubuntu2_i386.deb'
- type 'sync' to make sure that everything gets written for sure
- hit ctrl+alt+del to reboot. you should be set now.


jyio (inportb) wrote :

yep, i just spent hours figuring this out. in the end, i just did a clean install, then fixed my /etc/apt/preferences to say:

Package: libc6
Pin: version 2.7-5ubuntu2
Pin-Priority: 1001

And it's going to stay this way until libc6 is fixed :x

TheGZeus (gzeusmants) wrote :

Manually copying the contents of the older .deb files, as described in the forum thread, works perfectly.
Flawless victory.

Anders Kvist (akv) wrote :

Someone said su worked, not here, not even bash can be started... one bricked workstation :)

$ bash

malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
Aborting...complete: usage: complete [-abcdefgjksuv] [-pr] [-o option] [-A action] [-G globpat] [-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F function] [-C command] [name ...]

malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
Aborting...Aborted (core dumped)

Kyle M Weller (kylew) wrote :

ok, all u do is boot your live cd and open a terminal, wget
then mount your ubuntu drive whichever that is, it may be here:/media/disk-1 or something or another, open a terminal and then dpkg -x libc6_2.7-5ubuntu2_i386.deb /media/drivewhereubuntu is already installed then reboot

Lillipuziano (lillipuziano) wrote :

Here's how I did:

* Boot from the Live CD/DVD
* Depending on the Live CD you are using (Gnome is better, cause you'll only need to double click the appropriate icon on your desktop), you'll have to manually mount your root partition or not.
If you need to mount it (so, if you are using KDE), the best way is:

sudo su
mkdir /myroot
mount /dev/sda1 /myroot (where /dev/sda1 should be replaced in case)

* Then, download the old glibc:

chroot /myroot wget

* And run dpkg in order to downgrade it:

chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb

* After, you can always remove the package from your root dir.

the4dk (the4dk) wrote :

Thanks Lillipuziano! Your solution works perfect ;) I got my system back in few minutes ;)

Max Schukin (schukin) wrote :

Kyle M Weller, Lillipuziano, thanks a lot you for your solutions. Works great for me.

Fred Hermanns (fintan) wrote :

@jyio: What if you don't have a /etc/apt/preferences?

Just create one in kate, copy your lines into that new preferences and save it to /etc/apt and ?

Or how else?

Mikael Gerdin (mgerdin) wrote :

@Fred: that's what I just did, and "apt-cache policy libc6" now shows a higher priority for the currently installed version

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glibc - 2.7-9ubuntu2

glibc (2.7-9ubuntu2) hardy; urgency=low

  * Clear out LDFLAGS when building; glibc isn't happy building with
    -Wl,-Bsymbolic-functions. LP: #201673

 -- Steve Langasek <email address hidden> Thu, 13 Mar 2008 08:34:28 +0000

Changed in glibc:
status: Confirmed → Fix Released
Lillipuziano (lillipuziano) wrote :

How long does it take to new packages to appear in repos? I'm using the main server with Synaptic, but I still can't see it listed.

Bart Verwilst (verwilst) wrote :

Just give it a couple of hours. No pressure ;)

Colin Watson (cjwatson) wrote :

It will take a little while for this to become available, and it will probably require manual handholding on the buildds. All the necessary people are awake and aware of the problem, and we'll ensure that a response is available as quickly as we can.

Lillipuziano (lillipuziano) wrote :

Thanks guys! We all love Ubuntu. :-)

Marco Rodrigues (gothicx) wrote :

I had an open terminal when this happen and just have done:

$ wget

and after:

$ su -c "dpkg -i libc6_2.7-5ubuntu2_i386.deb"

And it works fine now.. :-) scared update :(

Bremm (bremm) wrote :

People are concerned about 'sudo' and MALLOC_CHECK_ variable. Anybody could run anything exporting it with "=1", but suid and sgid binaries don't work with it.

From `info malloc`:

There is one problem with `MALLOC_CHECK_': in SUID or SGID binaries it could possibly be exploited since diverging from the normal programs behavior it now writes something to the standard error descriptor. Therefore the use of MALLOC_CHECK_' is disabled by default for SUID and SGID binaries. It can be enabled again by the system administrator by adding a file `/etc/suid-debug' (the content is not important it could be empty).

It means I could 'touch /etc/suid-debug'. FYI: this bug caught me too; I'm an idiot, because my root has no password. lol

TheGZeus (gzeusmants) wrote :

Lillipuziano, Kubuntu mounts volumes automatically. One click in Dolphin, D3lphin or (either)Konqi on the disk in the Storage Media area.
Appearing on the desktop without an event (insertion of some media) doesn't work yet (at least in KDE3, I haven't spent much time with KDE4 yet).

Fred (eldmannen+launchpad) wrote :

I could not get your solution to work.

- root@ubuntu:/home/ubuntu# chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb
- dpkg-deb: --extract needs a target directory.
- Perhaps you should be using dpkg --install ?

Colin Watson (cjwatson) wrote :

Fixed packages are now available on

Cleber Santz (clebersantz) wrote :

Me too....

I reinstall the system, and waiting for stable update

André Lemos (alemos) wrote :

Fred, it's:

dpkg -X package /target/directory/

(notice the capital X)

Fred (eldmannen+launchpad) wrote :

Okay, got my system fixed now.
Using the LiveCD and;
cp --parents `dpkg -L libc6-i686 | xargs` /path/to/mounted/disk/
cp --parents `dpkg -L libc6 | xargs` /path/to/mounted/disk/


Shouldn't important and system-critical updates like this have been put into 'hardy-proposed' (Pre-release updates) for a few days before it got put into 'hardy-updates' (Recommended updates) ?


On Thu, Mar 13, 2008 at 02:02:54PM -0000, Fred wrote:
> Lillipuziano,
> I could not get your solution to work.
> ----
> - root@ubuntu:/home/ubuntu# chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb
> - dpkg-deb: --extract needs a target directory.
> - Perhaps you should be using dpkg --install ?

Yes, -i worked for me.


   /V\ Jan Groenewald
  /( )\

We don't do that for development releases, no; hardy-updates only becomes useful after release. As is prominently mentioned in a number of places, you should not be using pre-release versions of Ubuntu unless you are comfortable dealing with occasional (even severe) breakage.

description: updated
Colin Watson (cjwatson) on 2008-03-13
description: updated
Colin Watson (cjwatson) on 2008-03-13
description: updated
ichudov (igor-chudov) wrote :

OK, stuff happens. I recovered my laptop and will recover my server that is being configured for production. While I did not expect bugs that severe, I was warned and recovery was not that difficult. No biggie.

Here's three different conclusions:

1) R o o t P a s s w o r d

What is apparent to me, is that many system recovery features do require having a valid unix root password.

I think that it is highly precarious to leave users unable to "just screw that fancy stuff and logon as root" as they would be unable to perform various system recovery functions if that is so. (example, /home directory lost)

After any Ubuntu install, my FIRST command is always

  sudo passwd root

I think that a suitable installation action would be to "set root password to the first user password". (my root password is not the same)

2) M a i l i n g L i s t

I would personally like to subscribe to a "hardy-alerts" mailing list and get notified of such things as they occur. If such a list exists, I would like to subscribe to it, if not, I would like to see it. Not for minor bugs, but for showstopper bugs.

3) R e g r e s s i o n T e s t i n g

I write regression tests for anything that I program at work. I think that in the case of Ubuntu, it is possible to come up with an automated test that involves, at the very least, a successful multiuser boot, user able to login, ssh working, and the "system update drivetrain" working as well. A minimum system config that does not leave its users stranded. Such a test would be even more useful for production stage. If anyone needs help here, I could dedicate some time to it.

Bremm (bremm) wrote :


I guess I did a miracle. lol

WARNING: if you want try this procedure, read with attention the whole thing before try. It shouldn't harm your system, but need to be careful to avoid screw it up and need redo everything again. This nano-howto is newbie-friendly and released under GPLv2. lmfao

Let me explain exactly what I did... As you can see above, I told my root doesn't have password and suid/sgid binaries and doesn't work with MALLOC_CHECK_="1". But I'm running Xubuntu on a amd64 box, which means I need some stuff functional on 64 and 32 bits (thanks Adobe for beeing lazy with flash plugin). Ehr, so I have ia32libs and also dchroot here. You *need* at least a xterm/console/whatever already running and a web browser would help a lot. Now the fancy workaround:

* run 'dhcroot -d' (the new environment doesn't matter, but I'm assuming you already used it once)

* run 'mkdir newmountpoint' (or any name, just need to be unique)

* run 'sudo mount /dev/sda1 newmountpoint' (check before what's the right partition)

* now 'cd newmountpoint/etc' (check if the content is the same at regular /etc with no chroot)

* create an empty file with 'sudo touch suid-debug' (and check if this file is under /etc, same as above)

* go back with 'cd' and run 'umount /dev/sda1' (might check the right partition above)

* close the chroot running 'exit' (JUST ONCE, BE CAREFUL)

Now you are again on buggy console. Download the new libc6_2.7-9ubuntu2_amd64.deb from main repository, as explained by Marco Rodrigues (using wget) or some browser. If your system is 32 bits download the right i386 package. Now you'll do the easy part:

* run 'export MALLOC_CHECK_=1'

* and run 'sudo dpkg -i libc6_2.7-9ubuntu2_amd64.deb'

Whew! If you did all steps right, you'll see something like below (it's Portuguese, i forgot to use LANG=C to get English text, anyway you dont need it too):

malloc: using debugging hooks
*** glibc detected *** sudo: free(): invalid pointer: 0x00007f88ac9e66c8 ***
*** glibc detected *** sudo: free(): invalid pointer: 0x00007f88ac9e6710 ***
(Lendo banco de dados ... 169973 arquivos e diretórios atualmente instalados.)
Preparando para substituir libc6 2.7-9ubuntu1 (usando libc6_2.7-9ubuntu2_amd64.deb) ...
Descompactando substituto libc6 ...
Instalando libc6 (2.7-9ubuntu2) ...

Hope that helps. (:

Isma (isma-af) wrote :

Thanks Bremm for your hint!!

Sudo and su -c weren't working for me until I did "export MALLOC_CHECK_=1"
After that, I have downgraded the package and everything is working again now :D

Colin Watson (cjwatson) on 2008-03-13
description: updated
nicobrainless (nicoseb) wrote :

O maaaaan, you just saved me from rebooting... that is sweet as!!! (I didn't know about '-c' and it doesn't work without it!!)

>Marco Rodrigues wrote 8 hours ago: (permalink)
>I had an open terminal when this happen and just have done:
>$ wget
>and after:
>$ su -c "dpkg -i libc6_2.7-5ubuntu2_i386.deb"
>And it works fine now.. :-) scared update :(

Elias K Gardner (zorkerz) wrote :

following the procedure without a liveCD in the description I get an error after the 'mount -o remount,rw /root'

mount: Can't find /root in /proc/mounts

do i have any other options besides using a live cd?

TGM (tommann.home) wrote :

Just to check your mirrors:
run 'sudo apt-get update'
run 'sudo apt-get -s upgrade | grep libc6'

If you get the following (note the 2.7-9ubuntu2) you should be good to go

  libavcodec1d libavutil1d libc6 libcairo2 libcupsimage2 libcupsys2 libdb4.6
Inst libc6 [2.7-5ubuntu2] (2.7-9ubuntu2 Ubuntu:8.04/hardy)
Conf libc6 (2.7-9ubuntu2 Ubuntu:8.04/hardy)

If you have 2.7-9ubuntu1 do not upgrade just yet :)

Colin Watson (cjwatson) wrote :

I've contacted Elias by private e-mail to try to diagnose his problem. If it turns out to be general I'll update the instructions here.

Francisco Maia (amaia) wrote :

I just reinstalled my hardy laptop because of this.
I tried a lot of stuff without luck.
Since I had no Internet access to check this page, which could had saved me the reinstalation.

Tough luck...

Peter (peter-ashbourne) wrote :

Not 100% sure this is related. I'm running Hardy Alpha 64-bit, not because I am experienced, but because it has the drivers necessary for my wi-fi that Gutsy lacked.

Very successful until recent upgrade which failed to complete. The system is still booting and working fine but it impossible to do any installs/updates/upgrades.

Adept Updater simply reports
There was an error commiting changes. Possibly there was a problem downloading some packages or the commit would break packages.

Synaptic finds two broken packages: libc6-dev and libc6-i386, claims to successfully fix the dependency problems, but clearly doesn't!

libc6 is marked for upgrade from 2.7-5ubuntu2 to 2.7-9ubuntu2 but the upgrade fails and cannot be "unmarked".

Report from applying the upgrade is:
E: /var/cache/apt/archives/libc6_2.7-9ubuntu2_amd64.deb: subprocess pre-installation script returned error exit status 1

sudo apt-get -f upgrade gives (install gives similar):

Preparing to replace libc6 2.7-5ubuntu2 (using .../libc6_2.7-9ubuntu2_amd64.deb) ...
debconf: DbDriver "config": could not open /var/cache/debconf/config.dat
dpkg: error processing /var/cache/apt/archives/libc6_2.7-9ubuntu2_amd64.deb (--unpack):
 subprocess pre-installation script returned error exit status 1
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)

Any help appreciated, I do realise that Alpha versions carry risks, so will live with it if I have to.

Colin Watson (cjwatson) wrote :

Peter: this is not directly related to this bug. It looks like the directory /var/cache/debconf is entirely missing, perhaps due to some filesystem corruption. Feel free to contact me by private mail (<email address hidden>) and I'll try to help you out more.

Tom Arnold (g0tt) wrote :


Why dont you run updates for important base packages through a virtualbox installation of Ubuntu first. If it breaks you can just go back to save point X and fix it. Nobody will ever know ;)

The process could even be automated.
Only send the update if the vbox installtion reboots and shows X11 (gdm, kdm .. ) || login ( for a server ).

I think people do test/run LTS releases even in development. And this is not what i would call a confidence building exercise ;)
And we exspect bugs and breakage .. but not easily avoidable severe breakage :-P

But what do i know .. i just wanted to give my .02€

Colin Watson (cjwatson) wrote :

We are investigating what happened here (in terms of processes) and will make testing improvements as a result. I don't want to go into details prematurely.

Of course, resource-hungry testing measures are always a trade-off between development efficiency and safety. Ubuntu has derived great benefit in the past from being able to push uploads through quickly (so that many iterations can be run over the course of a working day), and I'm cautious about imposing mandatory delays that would slow that process down too badly. There are clearly much more lightweight measures that would achieve significant safety improvements, and we're likely to look at those first.

We will probably also ramp up our warnings about the development release not being suitable for production use. The object of the development release, whether leading up to an LTS or not, is not to build confidence; it is to produce a high-quality end result - and I do believe that the root cause of this change (LDFLAGS=-Wl,-Bsymbolic-functions) will provide significant benefits elsewhere. Yes, it is a shame that it blew up here, but on the bright side this is far enough before release that we have time for intensive testing to ensure that nothing else important is going to blow up in the same way.

It's very common for the reaction to failures to be to impose a much greater testing load on developers and/or development processes. You can increase testing to an unbounded extent, though, and clearly you hit diminishing returns at some point; even at that point, in our imperfect world, you'll still get failures. Thus we need to consider trade-offs and efficiency rather than simply saying "more testing please"; rest assured that that consideration will happen.

I do not want to discuss testing further on this bug report. Please take further discussion to development mailing lists.

Bart Verwilst (verwilst) wrote :

It's a development build. Don't like breakage, then don't use it, it's as simple as that. People using Hardy are ( or at least should be! ) aware that such things can occur, so they should deal with it accordingly, period.

Jan-Marc (jmvdb) wrote :


I had a problem after upgrading from 7.10 to 8.04. Got the message: "Waiting for root file system". Below my way to solve/work around this issue.

I did boot the system with an older kernel, you can choose from GRUB.
After logging in I installed "live-initramfs" using the package manager.
It seems that it has reconfigured initramfs. Rebooting again and the
systems boots well.

Colin Watson (cjwatson) wrote :

Jan-Marc: live-initramfs is not appropriate, though it may have happened to fix this by accident. A more correct fix is just to run 'update-initramfs -u' as root, as my (updated) instructions in the description of this bug indicate.

NoOp (glgxg) wrote :

"first reported on 2008-03-13'

Must have something to do with the 13th... Just did today's upgrades (April 13th) and the linux-restricted 386 kernel drops to initramfs busybox shell. Booting into -generic works fine.

Oddly enough, installing live-initramfs in the working (generic) kernel did not resolve the problem. But then removing (sudo apt-get remove live-initramfs) did, as the removal regenerated new boot initrd.img's for all the installed kernels.

$ sudo apt-get remove live-initramfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  user-setup busybox
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 373kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 150602 files and directories currently installed.)
Removing live-initramfs ...
update-initramfs: Generating /boot/initrd.img-2.6.24-16-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-16-386
update-initramfs: Generating /boot/initrd.img-2.6.24-15-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-15-386
update-initramfs: Generating /boot/initrd.img-2.6.24-14-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-14-386

To post a comment you must log in.