After LTS->LTS (lucid2precise) upgrade, upon reboot drops into grub recovery shell

[Jean-Yves Avenard]

[Impact]
Any system upgraded from Lucid->Precise is affected, if and only if the user does not opt-in to reinstall grub-pc bootloader during the do-release-upgrade/dist-upgrade nor reinstalls grub-pc into MBR manually before mandatory reboot.

The affected system will fail to reboot, and will drop into grub shell requiring manual boot.

[Workaround]
Users should follow the warnings about potential incompatibilities, and reinstall grub-pc into MBR/partitions as appropriate. This will update their grub to a supported version. If users still preffers to keep old grub, we provide Development and Stable Fixes.

[Development Fix]
In the Development branch, the fix was to use backwards compatible syntax in grub.cfg. Specifically, using ${var} syntax for variable substitution. (NB! the $var syntax was introduced as fully supported after the grub2 1.98 release, that is after lucid)

[Stable Fix]
Patch can be found at: https://code.launchpad.net/~dmitrij.ledkov/ubuntu/precise/grub2/fix-978464/+merge/105844

[Test Case]
1) Install/Have already installed: grub-common grub-pc grub-pc-bin grub2-common from lucid (1.98-1ubuntu5)
2) Make sure version of 1.98-1ubuntu5 of grub2 is used as bootloader, e.g. run `sudo grub-install /dev/sdX # Example: sudo grub-install /dev/sda'

Note: step 1 & 2 can be accomplished by doing a fresh lucid install in environment which uses grub-pc, e.g. desktop or server images, but not ec2 "cloud" images.

3) Upgrade grub-common grub-pc grub-pc-bin grub2-common from precise-proposed (1.99-21ubuntu3.1) but do not reinstall grub into MBR. (as part of do-release-upgrade/dist-upgrade select "Yes, continue without reinstalling grub")
4) Reboot

Assert True: reboot succeeds.

To test that bug exists: install grub-common grub-pc grub-pc-bin grub2-common from precise (1.99-21ubuntu3) but dot reinstall grub into MBR. Rebooting should drop into grub recovery console.

[Regression Potential]
Regressions are not anticipated, since only variant of valid syntax is changed. Nonetheless users should see https://help.ubuntu.com/community/Grub2 if they are having troubles booting their system.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/978464/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Jean-Yves Avenard (jyavenard)]

I should probably add that the existing 10.04 image was an upgrade from 8.04, and was likely still using GRUB 0.96 previously.

The target is a VMware VM, that I used to mirror existing setup and try upgrades beforehand

[Jean-Yves Avenard (jyavenard)]

to make my system boot again I had to do:
sudo rm /boot/grub/grub.cfg
sudo update-grub

[Thomas Mashos (tgm4883)]

I've recently tested a Mythbuntu 10.04 install (fresh install in a VM, fully updated) to 12.04 using 'update-manager -d'. Same issue. I received an error during upgrade stating that it wasn't set to install grub

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ante Karamatić (ivoks)]

It's probably the same issue. During an upgrade, I've got asked if I wish to continue without installing grub. Since answering No would result in broken upgrade, I've answered yes. On reboot, grub dropped to a shell. It was obvious that this was grub from 10.04 (1.98) and that upgrade never installed new grub into MBR. Installing grub with 'install-grub /dev/vda' solved the problem. Note that this is virtualized system; not sure if that changes anything.

[Brian Murray (brian-murray)]

It'd be helpful if we could get somebody's upgrade log files which exist in /var/log/dist-upgrade. We'll need all of them.

[MarcRandolph (mrand)]

sorry for the delay - been working non-stop for 7 days. Attached are the logs from BEFORE the reboot where you end up at the grub prompt. This is done in a VM. I also captured /boot/grub

[Brian Murray (brian-murray)]

I believe this is where things went wrong:

Transitioning /etc/network/if-up.d/upstart to be a conffile ...
Transitioning /etc/init/network-interface.conf to be a conffile ...
.2-2ubuntu30_i386.deb) ...
Transitioning /etc/init/networking.conf to be a conffile ...
Unpacking replacement ifupdown ...
f-13.10ubuntu11_i386.deb) ...
dpkg: considering deconfiguration of friendly-recovery, which would be broken by installation of upstart ...
dpkg: yes, will deconfigure friendly-recovery (broken by upstart).
Preparing to replace upstart 0.6.5-6 (using .../upstart_1.5-0ubuntu5_i386.deb) ...
De-configuring friendly-recovery ...
Unpacking replacement upstart ...
Preparing to replace friendly-recovery 0.2.10 (using .../friendly-recovery_0.2.25_all.deb) ...
Unpacking replacement friendly-recovery ...
Processing triggers for man-db ...
.deb) ...
Processing triggers for ureadahead ...
Setting up libdbus-1-3 (1.4.18-1ubuntu1) ...
Setting up libnih1 (1.0.3-4ubuntu9) ...
Setting up libnih-dbus1 (1.0.3-4ubuntu9) ...
Setting up libudev0 (175-0ubuntu9) ...
_i386.deb) ...
Setting up insserv (1.14.0-2.1ubuntu2) ...
Installing new version of config file /etc/insserv.conf ...
Setting up sysv-rc (2.88dsf-13.10ubuntu11) ...
Setting up libdrm2 (2.4.32-1ubuntu1) ...
Setting up libplymouth2 (0.8.2-2ubuntu30) ...
dpkg: dependency problems prevent configuration of ifupdown:
 ifupdown depends on upstart-job; however:
  Package upstart-job is not installed.
  Package upstart which provides upstart-job is not configured yet.
 ifupdown depends on initscripts (>= 2.88dsf-13.3); however:
  Package initscripts is not configured yet.
dpkg: error processing ifupdown (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin ...
No apport report written because the error message indicates its a followup error from a previous failure.
                          ldconfig deferred processing now taking place

[Pausanias (pausanias)]

I am affected by this same bug. Also within the context of VMWare.

Thank God for the ability to rollback to an older snapshot!

[Hans Deragon (deragon)]

Suffered from the same problem in my VM which initially was created with a fresh installation of 10.04. I reverted back my VM and I have not kept the broken one for its logs, unfortunately.

[Steve Langasek (vorlon)]

It's very strange that all of these reports are in VMs. Is there some common VMware support package that's involved here?

Has anyone reproduced this problem with the final release of 12.04 using update-manager (do-release-upgrade)?

[Steve Langasek (vorlon)]

Marc, you're also the submitter of bug #937196, which seems to have the same issue in the logs and no one else appears to be reporting this exact problem (that we've seen confirmed by logs so far). The two bugs don't appear to be from the same machine however, as the list of installed packages is different. The trigger is likely one of the packages you have installed that are not part of a standard install.

[MarcRandolph (mrand)]

Steve, I start with Mythbuntu 10.04 in most cases. I believe Bug
#937196 was done on a real machine, but I'd have to go back and check.
 If I get a chance tonight, I'll try to double check.

[Hunter Allen (hunter-allen)]

I upgraded from my 10.04.4 LTS to 12.04 using the update-manager, and the same problem occurs. I am stuck in a grub recovery shell... The version is 1.98-1ubuntu12 if that helps. Any more info is available upon request! Just let me know how I can help!

[Seiichi Horie (registratnon)]

Hi

I have upgraded from 10.04 LTS to 12.04 by update-manager. And had same problem. This problem is able to reproduce 100% on my VMware Workstation 8.0.3.

By using Super Grub2 Disk, I could start the upgraded 12.04 and install-grub command could fix this.

I will upload my /var/log/dist-upgrade

[Seiichi Horie (registratnon)]

This is the attachement of post #19.

[Steve Langasek (vorlon)]

Seiichi, your upgrade log does not show the same problem seen in Marc's. You may have both ended up with grub not properly configured at the end of the upgrade, but the reasons for this seem to be different.

[David Pashley (david-davidpashley)]

After a little bit of experimenting (and lots of rebooting) I tracked down the problem to a number of syntax errors in the generated grub.cfg. See attached for the changes that allowed my VM to boot. Obviously the changes will need to be made to the source files rather than the grub.cfg, but these should be easy to make from the diff supplied.

(It would be great if grub would output a better error message when it fails to read its config file correctly)

[David Pashley (david-davidpashley)]

It's also worth mentioning that my running grub is 1.98-1ubuntu12, and 1.99-21ubuntu3 is installed, so obviously something during the upgrade fails to install the new version of grub.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Diff to my grub.cfg" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Steve Langasek (vorlon)]

On Sat, May 05, 2012 at 12:41:26PM -0000, David Pashley wrote:
> It's also worth mentioning that my running grub is 1.98-1ubuntu12, and
> 1.99-21ubuntu3 is installed, so obviously something during the upgrade
> fails to install the new version of grub.

This is normal practice. We don't upgrade the grub on disk when upgrading
the package.

[Steve Langasek (vorlon)]

So from David's diff, it appears the grub in lucid did not support $var syntax consistently, it requires us to use ${var}. I'm not sure what the rules are for when $var is supported and when it isn't. Perhaps it's only when quoting the variable that we have to use the latter syntax?

Since we know we aren't going to upgrade the MBR when upgrading from 10.04 to 12.04, we should make sure our grub.cfg remains compatible with older grub2.

[Phillip Susi (psusi)]

Huh? Whenever you upgrade grub the MBR is reinstalled. If that isn't happening, then that is the bug.

[Steve Langasek (vorlon)]

On Mon, May 07, 2012 at 05:41:43PM -0000, Phillip Susi wrote:
> Huh? Whenever you upgrade grub the MBR is reinstalled.

This has never been the case because we don't believe we have a reliable
enough way to identify the correct device to write to.

> If that isn't happening, then that is the bug.

Regardless, this is not behavior that it's reasonable to change in an SRU.
We need to fix the grub.cfg to be compatible with lucid grub.

[Phillip Susi (psusi)]

On 5/7/2012 3:28 PM, Steve Langasek wrote:
> This has never been the case because we don't believe we have a reliable
> enough way to identify the correct device to write to.

It has been the case for some time now. We use debconf to store the
devices that need grub reinstalled on upgrade. You can run
dpkg-reconfigure grub-pc to update/correct the list.

> Regardless, this is not behavior that it's reasonable to change in an SRU.
> We need to fix the grub.cfg to be compatible with lucid grub.

If you upgrade to precise, you should no longer be running the precise
grub, not lucid grub. The grub ABI is not stable, so whenever a new
grub package is installed, the MBR/core image must also be updated or
breakage will happen when it tries to use the updated modules in
/boot/grub. The same rationale applies to the config file.

[Steve Langasek (vorlon)]

On Mon, May 07, 2012 at 09:14:53PM -0000, Phillip Susi wrote:
> On 5/7/2012 3:28 PM, Steve Langasek wrote:
> > This has never been the case because we don't believe we have a reliable
> > enough way to identify the correct device to write to.

> It has been the case for some time now. We use debconf to store the
> devices that need grub reinstalled on upgrade. You can run
> dpkg-reconfigure grub-pc to update/correct the list.

Right - Colin set me straight on this. Sorry for the confusion.

> > Regardless, this is not behavior that it's reasonable to change in an SRU.
> > We need to fix the grub.cfg to be compatible with lucid grub.

> If you upgrade to precise, you should no longer be running the precise
> grub, not lucid grub. The grub ABI is not stable, so whenever a new
> grub package is installed, the MBR/core image must also be updated or
> breakage will happen when it tries to use the updated modules in
> /boot/grub. The same rationale applies to the config file.

Yes. But AIUI there are always some cases where we cannot upgrade grub on
the MBR for one reason or another, and this seems to be one of them - so we
really ought to preserve format compatibility where possible.

Dave, can you speak to why grub wouldn't have been upgraded in your case?
What's the output of 'debconf-show grub-pc' for you on the affected machine?

[Stephen Rees-Carter (valorin)]

I just did an upgrade on a VMware machine from 10.04 to 12.04 today.
Same issue here...

Can someone talk me through the process of fixing it?
I've never fixed an GRUB issue before, so I don't know what to do...

[Jean-Yves Avenard (jyavenard)]

Read comment #5 https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/978464/comments/5

[Dimitri John Ledkov (xnox)]

http://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr/revision/2260

revno: 2260 [merge]
committer: BVK Chaitanya <email address hidden>
branch nick: grub
timestamp: Sun 2010-03-14 22:37:17 +0530
message:
  2010-03-14 BVK Chaitanya <email address hidden>

   GRUB shell lexer and parser improvements.

1.98 was released on 2010-03-06.

Upgrades from Lucid -> Precise are affected.
Quantal is not affected, per se. Unless the user does not update grub after upgrading.
Debian squeeze and up are not affected (it has newer snapshot 1.98+20100804-14+squeeze1)

I would only do Precise SRU. But we could add the fix into quantal as well.

[Steve Langasek (vorlon)]

Whatever issue prevented the users from upgrading their MBR from 10.04 to 12.04 is unlikely to magically resolve itself for 12.10; so we should really fix this for quantal as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 1.99-21ubuntu4

---------------
grub2 (1.99-21ubuntu4) quantal; urgency=low

  * Switch to backwards compatible ${} syntax for grub.cfg (LP: #978464)
 -- Dmitrijs Ledkovs <email address hidden> Tue, 15 May 2012 08:03:30 -0700

[Dimitri John Ledkov (xnox)]

Ok.

For the record during Lucid -> Precise upgrade there are warnings shown recommending to re-install grub into MBR & it offers to select which drives/partitions to auto-install grub onto from now on. User may not know where grub is currently installed. It may be better to pre-select drives/partitions which currently have grub installed in that debconf question.

But this is besides the point. The two branches proposed resolve the issue. I will request merge proposals into correct branches now.

[Dimitri John Ledkov (xnox)]

* fix released in development release
* fix uploaded into -proposed (pending approval)
* description updated with details
* ubuntu-sru team subscribed

[Phillip Susi (psusi)]

That's not really a fix Dmitrijs. The problem stems from having an old version of grub in the mbr and boot track, and a new version in /boot. This will ( and appears to be from the number of bug reports ) cause multiple problems. The fact that the old grub doesn't like the new grub.cfg is the least of the problems.

[Steve Langasek (vorlon)]

On Tue, May 15, 2012 at 06:59:35PM -0000, Phillip Susi wrote:
> That's not really a fix Dmitrijs.

It's the fix I asked him to implement. As explained in the earlier
comments, there is no way we can guarantee with 100% reliability that the
MBR will be upgraded when the package is upgraded. We should therefore
maintain maximum compatibility here with old versions of grub - there's no
reason not to.

[Phillip Susi (psusi)]

Nevermind, I see now... in this case neither the mbr *nor* /boot have been upgraded.

[Chris Halse Rogers (raof)]

Hello Jean-Yves, or anyone else affected,

Accepted grub2 into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Stéphane Graber (stgraber)]

Bug confirmed with lucid => precise, now testing lucid => precise-proposed to confirm it's fixed.

[Stéphane Graber (stgraber)]

Fix confirmed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 1.99-21ubuntu3.1

---------------
grub2 (1.99-21ubuntu3.1) precise-proposed; urgency=low

  * Switch to backwards compatible ${} syntax for grub.cfg (LP: #978464)
 -- Dmitrijs Ledkovs <email address hidden> Tue, 15 May 2012 09:03:27 -0700

[Scott Ritchie (scottritchie)]

I think the releasing of the update may have exposed another bug, which counts as a regression: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1009294