upgrade of dbus causes unclean unmount of /

[Zachary Lassiter]

Running on Amazon Web Services AMI ami-057bcf6c (which is the official release from Ubuntu Availalbe at http://uec-images.ubuntu.com/releases/precise/release-20120822/ )

upon first start run

sudo apt-get update
sudo apt-get upgrade

it will tell you a restart is required when you next log in via ssh

restart the server

try to ssh in again and your keyfile will not work anymore

This is a major problem for AWS users as we can't access a console and there is no way to reset the keyfile and thus you are locked out of your server for ever. I've reproduced this once on a separate instance.

[Zachary Lassiter (7jc2)]

sorry if this is not a good bug report... my first bug report here.

[Zachary Lassiter (7jc2)]

after you do the apt-get update and apt-get upgrade if you log out on the motd it says reboot required... you won't see it unless you reboot it. Well I actually rebooted it now and its not doing it now if i install other packages.

odd.

[Scott Moser (smoser)]

Perhaps you could give some more information on what you did.
I just did this, and was unable to reproduce:
$ euca-run-instances ami-057bcf6c --instance-type m1.small --key brickies
$ ssh ubuntu@$HOSTNAME

% sudo apt-get update
% sudo apt-get upgrade
% sudo reboot

# note, In an attempt to guarantee a way in, I first added another user and set up ssh keys for that user, but then went in with the ubuntu user.

I then went back in and did:
 % sudo apt-get dist-upgrade

and everything was fine.

Please provide the instance type you were running with and anything you did between launch and reboot.

[Scott Moser (smoser)]

And also please provide a console log after the failed reboot. Note, console log in ec2 is delayed by 5 minutes, so please wait 5 minutes after you've rebooted, and then even terminate the instance, and wait 5 more and the collect the log. (I just want to make sure we dont lose anything that would be of value).

[Scott Moser (smoser)]

OK. I just tried to reproduce again, by
 * launching a m1.small, and t1.micro of ami-057bcf6c in us-east-1 in each availability zone that was available (for me that was 'us-east-1b', 'us-east-1c' and 'us-east-1e', but AZ names differ per account).
 * ssh in and 'apt-get update && apt-get upgrade'
 * reboot
 * ssh in again and 'apt-get dist-upgrade'
 * reboot
 * ssh in

I was unable to reproduce any failure.
However, I did see the following:
$ egrep "(/ is busy|INFO: recovery)" i-*
i-13208a6e.txt:mount: / is busy
i-13208a6e.txt:[ 1.650366] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
i-2f208a52.txt:mount: / is busy
i-2f208a52.txt:[ 1.032102] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
i-41208a3c.txt:mount: / is busy
i-41208a3c.txt:[ 2.027800] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
i-45208a38.txt:mount: / is busy
i-45208a38.txt:[1379725.446288] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
i-f3208a8e.txt:mount: / is busy
i-f3208a8e.txt:[819720.914345] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
i-fd208a80.txt:mount: / is busy
i-fd208a80.txt:[304615.135311] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem

Basically / was not unmounted cleanly on the first reboot. I suspect that the issue was that automatic fsck failed in the case the reporter saw. The attachment here contains all the console logs, as well as apt history from one of the systems.
For easy viewing, apt-history.log is below. It would seem like one of the packages in the *first* list caused a busy root filesystem.

Start-Date: 2012-10-01 13:18:28
Commandline: apt-get upgrade --assume-yes -q
Upgrade: bind9-host:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), libapt-pkg4.12:i386 (0.8.16~exp12ubuntu10.2, 0.8.16~exp12ubuntu10.3), dnsutils:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), libdbus-1-3:i386 (1.4.18-1ubuntu1, 1.4.18-1ubuntu1.1), grub2-common:i386 (1.99-21ubuntu3.1, 1.99-21ubuntu3.4), apt-transport-https:i386 (0.8.16~exp12ubuntu10.2, 0.8.16~exp12ubuntu10.3), libdns81:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), apport:i386 (2.0.1-0ubuntu12, 2.0.1-0ubuntu13), grub-pc:i386 (1.99-21ubuntu3.1, 1.99-21ubuntu3.4), libisccc80:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), isc-dhcp-client:i386 (4.1.ESV-R4-0ubuntu5.2, 4.1.ESV-R4-0ubuntu5.5), apt-utils:i386 (0.8.16~exp12ubuntu10.2, 0.8.16~exp12ubuntu10.3), dbus:i386 (1.4.18-1ubuntu1, 1.4.18-1ubuntu1.1), apt:i386 (0.8.16~exp12ubuntu10.2, 0.8.16~exp12ubuntu10.3), liblwres80:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), ubuntu-keyring:i386 (2011.11.21, 2011.11.21.1), libxml2:i386 (2.7.8.dfsg-5.1ubuntu4.1, 2.7.8.dfsg-5.1ubuntu4.2), libbind9-80:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), python-problem-report:i386 (2.0.1-0ubuntu12, 2.0.1-0ubuntu13), isc-dhcp-common:i386 (4.1.ESV-R4-0ubuntu5.2, 4.1.ESV-R4-0ubuntu5.5), grub-pc-bin:i386 (1.99-21ubuntu3.1, 1.99-21ubuntu3.4), libisccfg82:i386 (9.8.1.dfsg.P1-4ubuntu0.2, 9.8.1.dfsg.P1-4ubuntu0.3), tzdata:i386 (2012e-0ubuntu0.12.04, 2012e-0ubuntu0.12.04.1), gpgv:i386 (1.4.11-3ubuntu2, 1.4.11-3...

[Scott Moser (smoser)]

I've verified that I see the 'mount: / is busy' and "EXT4-fs (xvda1): INFO: recovery required on readonly filesystem" errors after upgrade of only libdbus-1-3 and dbus.

sudo apt-get update && sudo apt-get install dbus libdbus-1-3

[Scott Moser (smoser)]

I just did the following on a fresh instance:
 $ { echo dbus hold; echo libdbus-1-3 hold ; } | sudo dpkg --set-selections
 $ sudo apt-get update
 $ sudo apt-get upgrade --assume-yes --quiet 2>&1 | tee upgrade.log
 $ sudo reboot

There is no fsck or unclean mount on reboot.
So this would seem to fall squarely on dbus.

Interestingly, I followed the above with:
 $ { echo dbus install; echo libdbus-1-3 install ; } | sudo dpkg --set-selections
 $ sudo apt-get upgrade # pulling in the 2 dbus packages
 $ sudo apt-get remove dbus
 $ sudo reboot

and still saw the unclean mount.

[Scott Moser (smoser)]

Ok.
I can how reproduce this with:
 * launch instance
 * sudo stop dbus # and verify no dbus running
 * sudo apt-get update && sudo apt-get install libdbus-1-3
 * sudo reboot

I'm attaching 'before.txt' and 'after.txt' which contain
   ( set -x; sudo lsof +c15; sudo ps axww ) > FILE 2>&1
ran before and after the apt-get install.

[Steve Langasek (vorlon)]

until we have stateful reexec handling in upstart, dbus needs to set a flag to tell the system to re-exec upstart on shutdown so we can cleanly unmount (/var/run/init.upgraded). See the upstart postinst for details, and bug #441796 for history.

[Steve Langasek (vorlon)]

There was a recent security update of dbus in precise, which is why this bug is being noticed now. Security team, this should get an update in the security pocket. Can you look at this?

[Marc Deslauriers (mdeslaur)]

Is this only affecting precise images?

[Steve Langasek (vorlon)]

On Tue, Oct 02, 2012 at 05:21:18PM -0000, Marc Deslauriers wrote:
> Is this only affecting precise images?

It would affect all releases lucid and later. I didn't check which releases
besides precise had the security update.

[Scott Moser (smoser)]

It would appear that this is a duplicate of bug 740390.
Shall we dupe this to that?