libdbus-1-3 upgrade does not respawn init, resulting in unclean shutdown

[Alan]

Binary package hint: dbus

Steps to reproduce:
1. Start 20110321 daily build on Amazon EC2.
ami-30669459 ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321
2. SSH.
3. sudo apt-get update
4. To simulate an upgrade, sudo apt-get --reinstall install libdbus-1-3
libdbus-1-3_1.4.6-1ubuntu6 is reinstalled.
5. sudo lsof yields (snipped)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root DEL REG 202,1 134262 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
upstart-u 201 root DEL REG 202,1 134262 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
upstart-s 369 root DEL REG 202,1 134262 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
6. sudo shutdown -h now
Error:
mount: / is busy

Possible solution: touch /var/run/init.upgraded in package postinst to force init to respawn/close open files at shutdown. This is checked in /etc/rc[06].d/S60umountroot. See bug 672177 for a similar situation with libc6.

[Scott Moser (smoser)]

Ubuntu on EC2 project is not the right place to file these any more. Instead, please use 'ubuntu-bug <package>' on an instance, or tag with 'ec2-images' and 'uec-images'

[Scott Moser (smoser)]

I'm unable to reproduce this bug.

The ami given (ami-30669459 ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321) is no longer public.
I've tried running the latest daily :
  us-east-1 ami-d8e21eb1 canonical ubuntu-natty-daily-i386-server-20110418

then doing:
 $ sudo apt-get update
 $ sudo apt-get install --reinstall libdbus-1-3
 $ sudo shutdown -h now

I can see the '/ is busy' message in the console out, but the instance *does* go to terminated.

I suspect that the fallout of this is that the filesystem is not unmounted and thus not clean, right?

[Alan (8libra)]

>I suspect that the fallout of this is that the filesystem is not unmounted and thus not clean, right?

Correct. This results in orphaned inodes on reboot. Touching
/var/run/init.upggraded is a possible solution. This will cause init
to respawn on shutdown.

You have reproduced the bug. The problem isn't that the system doesn't
shut down. The problem is an unclean filesystem.

Thanks for starting to take a look.

On 4/18/11, Scott Moser <email address hidden> wrote:
> I'm unable to reproduce this bug.
>
> The ami given (ami-30669459
> ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321) is no longer
> public.
> I've tried running the latest daily :
> us-east-1 ami-d8e21eb1 canonical ubuntu-natty-daily-i386-server-20110418
>
> then doing:
> $ sudo apt-get update
> $ sudo apt-get install --reinstall libdbus-1-3
> $ sudo shutdown -h now
>
> I can see the '/ is busy' message in the console out, but the instance
> *does* go to terminated.
>
> I suspect that the fallout of this is that the filesystem is not
> unmounted and thus not clean, right?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/740390
>
> Title:
> libdbus-1-3 upgrade does not respawn init, resulting in unclean
> shutdown
>
> Status in Official Ubuntu Images on EC2:
> Invalid
> Status in “dbus” package in Ubuntu:
> New
>
> Bug description:
> Binary package hint: dbus
>
> Steps to reproduce:
> 1. Start 20110321 daily build on Amazon EC2.
> ami-30669459
> ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321
> 2. SSH.
> 3. sudo apt-get update
> 4. To simulate an upgrade, sudo apt-get --reinstall install libdbus-1-3
> libdbus-1-3_1.4.6-1ubuntu6 is reinstalled.
> 5. sudo lsof yields (snipped)
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> NAME
> init 1 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> upstart-u 201 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> upstart-s 369 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> 6. sudo shutdown -h now
> Error:
> mount: / is busy
>
> Possible solution: touch /var/run/init.upgraded in package postinst to
> force init to respawn/close open files at shutdown. This is checked in
> /etc/rc[06].d/S60umountroot. See bug 672177 for a similar situation
> with libc6.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu-on-ec2/+bug/740390/+subscribe
>

[Clint Byrum (clint-fewbar)]

I just tested this on a clean natty VM with all updates applied, and I do indeed see an orphaned inode left over on /

We really need to trigger /var/run/init.upgraded on any of upstart's dependent libraries. Rather than making all the libraries do this, a trigger on /lib seems to make more sense where it can do an 'ldd' on /sbin/init and if the libraries changed fall into it, touch /var/run/init.upgraded. So, I'll open a task against upstart to do that. In the interim, we need to have libdbus-1-3 touch /var/run/init.upgraded as well.

[Scott Moser (smoser)]

Well, hindsight being 20-20, I should have seen this as more serious. Bug 1058517 is now open, it is really a dupe of this. we should collapse them one way or another.

[Colin Watson (cjwatson)]

Clint: I don't think there's any point in doing a complex trigger arrangement when (a) only a tiny number of libraries are involved, and (b) stateful-reexec will solve this anyway.

FWIW, libnih1/libnih-dbus1 have this problem too.

[Steve Langasek (vorlon)]

Added tasks for libnih.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dbus - 1.6.4-1ubuntu3

---------------
dbus (1.6.4-1ubuntu3) quantal-proposed; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * Fix unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:14:40 -0400

[Clint Byrum (clint-fewbar)]

If stateful re-exec does indeed handle this then agreed, no need for any triggers.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dbus - 1.4.18-1ubuntu1.3

---------------
dbus (1.4.18-1ubuntu1.3) precise-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 06:12:39 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package dbus - 1.2.16-2ubuntu4.7

---------------
dbus (1.2.16-2ubuntu4.7) lucid-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:05:52 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package dbus - 1.4.6-1ubuntu6.4

---------------
dbus (1.4.6-1ubuntu6.4) natty-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:03:55 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package dbus - 1.4.14-1ubuntu1.3

---------------
dbus (1.4.14-1ubuntu1.3) oneiric-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:02:41 -0400

[dino99 (9d9)]

The dbus upgrade has been applied, and this basic unique desktop still show on shutdown: mount: / is busy (each time it is shutdown). This is on QQ i386 with pata/sata hdds and bios.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libnih - 1.0.3-4ubuntu11

---------------
libnih (1.0.3-4ubuntu11) quantal; urgency=low

  * Addition of debian/libnih-dbus1.postinst and
    debian/libnih1.postinst to force Upstart re-exec on shutdown
    to avoid unmounting disks uncleanly (LP: #740390).
 -- James Hunt <email address hidden> Wed, 03 Oct 2012 16:49:40 +0100

[James Hunt (jamesodhunt)]

@dino99: Thanks for spotting this - I too saw this in testing. I've isolated the problem - see bug 1061639.

[dino99 (9d9)]

Thanks James, confirmation done.

[James Hunt (jamesodhunt)]

I've created bug 1062202 to handle the lbnih SRU for precise.

[dino99 (9d9)]

Still get uncleaned shutdown process:

with the last shutdown, i've seen:
- modemmanager reloading its dozen drivers
- mount / is busy
- and last with a [drm] nouveau complaint, switching to fbcon (or so)

with the next reboot:

oem@dub:~$ dmesg | grep orphan
[ 8.812542] EXT4-fs (sdb5): orphan cleanup on readonly fs
[ 8.819511] EXT4-fs (sdb5): ext4_orphan_cleanup: deleting unreferenced inode 400809
[ 8.819581] EXT4-fs (sdb5): 1 orphan inode deleted

[Brian Murray (brian-murray)]

Hello Alan, or anyone else affected,

Accepted libnih into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libnih/1.0.3-4ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Colin Watson (cjwatson)]

I verified this following James' and Marc's instructions in bug 1062202.

[Colin Watson (cjwatson)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libnih - 1.0.3-4ubuntu9.1

---------------
libnih (1.0.3-4ubuntu9.1) precise-proposed; urgency=low

  * Addition of debian/libnih-dbus1.postinst and
    debian/libnih1.postinst to force Upstart re-exec on shutdown
    to avoid unmounting disks uncleanly (LP: #740390, LP: #1062202).
 -- James Hunt <email address hidden> Fri, 05 Oct 2012 14:14:10 +0100