Running update-grub-legacy-ec2 doesn't update kernel list in /boot/grub/menu.lst

[Peter Waller]

This bug has been observed with at least EC2 AMI: ami-ec50a19b "ubuntu/images/ebs/ubuntu-saucy-13.10-amd64-server-20140212", and other Ubuntu saucy AMIs.

Steps to reproduce:

1. sudo apt-get update
2. sudo apt-get dist-upgrade
3. Get prompted for one thing, whether to install the maintainer's version of /boot/grub/menu.lst (choose to use the maintainer's file)
4. observe updated file
5. modify the file to remove the new entries
6. run update-grub-legacy-ec2
7. observe file not updated

This is especially insidious because it claims the file is updated, with output like this:

> Found kernel: /boot/vmlinuz-3.11.0-19-generic
> Found kernel: /boot/vmlinuz-3.11.0-18-generic
> Found kernel: /boot/vmlinuz-3.11.0-17-generic
> Found kernel: /boot/vmlinuz-3.11.0-15-generic
> Found kernel: /boot/vmlinuz-3.11.0-14-generic
> Found kernel: /boot/vmlinuz-3.11.0-12-generic
> Found kernel: /boot/memtest86+.bin
> Updating /boot/grub/menu.lst ... done

And in fact, /boot/grub/menu.lst has its modified time updated to now, but it doesn't actually change the contents of the file.

This is extremely confusing.

I note that UCF_FORCE_CONFFNEW=1 and DEBIAN_PRIORITY=low and:

> # debconf-get-selections | grep grub-l
> grub-legacy-ec2 grub/update_grub_changeprompt_threeway select install_new

don't have any useful effect.

I've emailed Ben Howard who confirmed he could reproduce this bug.

[Ben Howard (darkmuggle-deactivatedaccount)]

I've confirmed that this happens on non-EC2 systems. I was able to replicate this on Canonistack (OpenStack) but not on EC2.

[Ben Howard (darkmuggle-deactivatedaccount)]

Peter,

Can you tell me more about how you are hitting this in EC2? Are you booting an HVM instance-store AMI and then running ec2-bundle-vol? The only way that I can see this happening is if you boot with BIOS/GPT (i.e. HVM instance-store) and then try to create a volume and expect it to boot using PVGrub (which uses legacy Grub menu.lst).

[Peter Waller (peter.waller)]

It's not a HVM instance. The AMI is supplied at the top of the bug report.

Note that this bug has nothing to do with which kernel is running on the machine after reboot, except that the file `/boot/grub/menu.lst` has the wrong contents.

The buggy behaviour is that `/boot/grub/menu.lst` *IS NOT BEING UPDATED* even though the message on the console actually says "Updating /boot/grub/menu.lst ... done".

To reproduce the behaviour all you have to do is modify `/boot/grub/menu.lst` once ever. Then it will never be updated again, even though `update-grub-legacy-ec2` claims it is updating the file.

Unless I'm missing something, it appears that the debconf setting is being ignored.

The net effect is that we don't get kernel updates without manually intervening on `/boot/grub/menu.lst`.

[Peter Waller (peter.waller)]

Please subtract "Note that this bug has nothing to do with which kernel is running on the machine after reboot, except that" from my last comment. I'm somewhat tired and I'm not sure what I meant by it.

Further digesting what you're saying, `/boot/grub/menu.lst` is not used for EBS AMIs? This doesn't seem consistent with what I've observed, which is that machines only boot to the new kernel if I modify this file manually.

It is possible that some other interaction is fooling me into thinking this, though, and it has been hard to pin down consistent behaviour.

It seems in any case that the buggy (or at least confusing) behaviour of update-grub-legacy-ec2 is problematic since it has wasted many man hours as we try and figure out why it refuses to update on some machines but not others.

[Peter Waller (peter.waller)]

I've just discovered the doing a `sudo rm /boot/grub/menu.lst` and `sudo update-grub-legacy-ec2` fixes the problem, at least temporarily.

[glance (glance-acc)]

This have started to affect me to.

Older machine based on ubuntu-trusty-14.04-amd64-server-20140416.1 isn't affected by this problem but newer machines as of ubuntu-trusty-14.04-amd64-server-20140607.1 are affected.

[Stratos Zolotas (baskin)]

I'm experiencing the same issue on an EC2 (AWS) Ubuntu-14.04 instance.

Although the following kernels are installed:

Found linux image: /boot/vmlinuz-3.13.0-32-generic
Found initrd image: /boot/initrd.img-3.13.0-32-generic
Found linux image: /boot/vmlinuz-3.13.0-30-generic
Found initrd image: /boot/initrd.img-3.13.0-30-generic
Found linux image: /boot/vmlinuz-3.13.0-29-generic
Found initrd image: /boot/initrd.img-3.13.0-29-generic

the system boots with 3.13.0-29 and the menu.lst is not updated.

[Peter Waller (peter.waller)]

For what it's worth, HVM machines appear to be unaffected.

It seems that the fix of deleting the menu.lst only works once must be deleted to update it subsequently.

[Anders Hall (a.hall)]

We have a similar bug in https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1323772 you might want to join.

[Cam Cope (ccope)]

This affects me on Ubuntu 12.04 with grub-legacy-ec2 0.7.5-0ubuntu1.12

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nathan Stratton Treadway (nathanst)]

I ran into a problem with grub-legacy-ec2 v1:1 in Bionic (on an EC2 instance built from a Canonical Bionic AMI)... but the problem I found relates to the switch from /var/run to /run, so it seems like it could be related the earlier reports as well.

Specifically, what I noticed was the "The first time ucf sees the file" check around line 1349 of update-grub-legacy-ec2, which is:

=====
# The first time ucf sees the file, we can only assume any difference
# between the magic comments and the kernel options is a result of local
# mods, so this will result in a ucf prompt for anyone whose first
# invocation of update-grub is as a result of updating the magic comments.
if ! ucfq grub | grep -q $ucf_menu_file; then
        otherbuffer=$(tempfile)
        cat $buffer > $otherbuffer

        sortedKernels=`sed -n -e "
        /$endopt/,/$end/ {
                s/^kernel[[:space:]]\+\([^[:space:]]\+\).*/\1/p
        }" < $menu | grep -vE "memtest86|$grub2name|xen" | uniq`
        xenKernels=`sed -n -e "
        /$endopt/,/$end/ {
                s/^module[[:space:]]\+\([^[:space:]]*vmlinuz[^[:space:]]\+\).*/\
1/p
        }" < $menu | uniq`

=====

When this is called, $ucf_menu_file contains "/var/run/grub/menu.lst"... but because of the /var/run -> /run switch, UCF doesn't actually have registered a file with that path:

=====
# ucfq grub
Configuration file Package Exists Changed
/run/grub/menu.lst grub Yes Yes
# if ! ucfq grub | grep -q /var/run/grub/menu.lst; then echo "yes"; fi
yes
=====

So the net effect is that the script assumes that the file has not been registered with UCF before and rebuilds it from the kernel list taken out of the existing /boot/grub/menu.lst... but since that list is stale, there is a ucf conflict and the new file is not installed -- and then later when the script tries to to update the file with the currently-installed kernel list, there is still a ucf conflict and the updated menu.lst file is not installed.

I was able to get past this by running the kernel-image install manually (using aptitude) and thus getting the ucf "A new version of /boot/grub/menu.lst is available, but the version installed currently has been locally modified" dialog box. When I chose the "install the package maintainer's version" version, the script was then able to proceed to install that (outdated) menu.lst file, and then went on to detect the current kernel list and update menu.lst reflect them...

(I have not yet tried a second kernel update to see if the problem reoccurs.)

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2444