Upgrade of glibc causes root filesystem not to be mounted ro on shutdown

[Guido Berhoerster]

After upgrading the glibc packages the root filesystem cannot be mounted ro by /etc/init.d/umountroot during the following shutdown/reboot, resulting in a "mount: / is busy" Error (see also attached screenshot). This is very BAD and may cause data loss.

I have tested this on fresh installed and otherwise unmodified X/K/Ubuntu 7.10 after an upgrade of the packages libc6 and libc6-i686 from version 2.6.1-1ubuntu9 to version 2.6.1-1ubuntu10 in gutsy-updates. It is easily reproducible in a VM as well as on physical hardware (see below).

Steps to Reproduce:
1. install any Ubuntu 7.10 flavour with default settings (in a VM or your hardware)
2. on the first boot use grub's edit function to remove the "quiet" and "splash" options from the kernel line so you can see what's going on during boot/shutdown
3. boot
4. upgrade the libc6 and libc6-i686 packages from version 2.6.1-1ubuntu9 to version 2.6.1-1ubuntu10
5. edit /etc/default/rcS anch change VERBOSE=no to VERBOSE=yes in order to make the init scripts' output more verbose
6. switch to a vt and reboot the system
7. at the end of the shutdown/reboot process observe the following error:
----8<----
 * Mounting root filesystem read-only...
mount: / is busy
                                                                        [fail]
---->8----

[Guido Berhoerster (gber)]

I made a small init script in order to see if there are any unusual processess with open filehandles blocking the remount, however I couldn't spot anything on first sight.

I used the following script:

----lsof----
#!/bin/sh

lsof | tee /root/lsof.out
sync; sync
----lsof----

and made it run just before umountfs and umountroot on shutdown/reboot (via "update-rc.d lsof start 35 0 6 .").

The output is attached.

[faithful (strangecode)]

I had the same problem. Verified on x86 with 32 or 64 bits on ext3, jfs, xfs.
I had seriuous file corruption on many machines.
The only workaround is:
1) update only libc6;
2) reboot from a live cd to fix the filesystem;
3) reboot from the hd;
4) reinstall all the packages;
5) update the the system.
It's a dangerous bug!

[Scott James Remnant (Canonical) (canonical-scott)]

Matthias - you reassigned this to sysvinit, did you do any checks to make sure this isn't something caused by glibc?

It randomly occurs to me that processes may be running with the old glibc linked in, and haven't been killed yet.

[Scott James Remnant (Canonical) (canonical-scott)]

Ah, there was an lsof attached:

init 1 root txt REG 8,5 88672 191281 /sbin/init
init 1 root mem REG 8,5 96619 /lib/tls/i686/cmov/libc-2.6.1.so (path inode=96299)

So the still-running process with the old glibc mapped in is init!

[Scott James Remnant (Canonical) (canonical-scott)]

Looks like glibc does nothing to restart Upstart, despite having code in there for sysvinit

So this is a glibc bug after all!

[Scott James Remnant (Canonical) (canonical-scott)]

Let's just call this an Upstart bug for not implementing "telinit u" :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package upstart - 0.3.9-7

---------------
upstart (0.3.9-7) intrepid; urgency=low

  * Implement "telinit u" by just sending Upstart SIGTERM with a slightly
    different patch than Fedora. LP: #188925.

 -- Scott James Remnant <email address hidden> Tue, 23 Sep 2008 09:01:09 -0700