[SRU] Fast installer - failure to install grub (UEFI mode)

[Larry Michel]

[Impact]
This causes machines that MAAS / curtin will fail to install to those machines that have UEFI enabled.

[Test Case]
1. Install maas
2. register a system with UEFI enabled.
3. Deploy Ubuntu. It will fail because curtin will fail to deal with UEFI.

[Regression Potential]
Minimal. This actually fixes a regression. This has been tested in various labs and proven to be working on production environments.

[Other Info]

After recent upgrade to 1.7, fast installer trusty deployment with /dev/fioa as boot device does not work. This is for UEFI mode. It worked prior to upgrade. Below is output cloud-init-output.log:
"
...
grub-install: error: /usr/lib/grub/modinfo.sh doesn't exist. Please specify --target or --directory.
failed to install grub!
Unexpected error while running command.
Command: ['install-grub', '--uefi', '/tmp/tmpBpbjAX/target', '/dev/fioa']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "
<< additional log goes here >>
 ...\nSetting up efibootmgr (0.5.4-7ubuntu1) ...\nSetting up grub-efi-amd64-bin (2.02~beta2-9ubuntu1) ...\nSetting up grub-efi-amd64 (2.02~beta2-9ubuntu1) ...\nReplacing config file /etc/default/grub with new version\nGenerating grub configuration file ...\nFound linux image: /boot/vmlinuz-3.13.0-35-generic\nFound initrd image: /boot/initrd.img-3.13.0-35-generic\ndone\nSetting up grub-efi-amd64-signed (1.34.1+2.02~beta2-9ubuntu1) ...\nSetting up sbsigntool (0.6-0ubuntu7) ...\nSetting up secureboot-db (1.1) ...\nCan't access efivars filesystem at /sys/firmware/efi/efivars, aborting\nSetting up shim (0.4-0ubuntu4) ...\nSetting up shim-signed (1.6+0.4-0ubuntu4) ...\nReplacing config file /etc/default/grub with new version\nGenerating grub configuration file ...\nWarning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.\nFound linux image: /boot/vmlinuz-3.13.0-35-generic\nFound initrd image: /boot/initrd.img-3.13.0-35-generic\ndone\nGenerating grub configuration file ...\nWarning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.\nFound linux image: /boot/vmlinuz-3.13.0-35-generic\nFound initrd image: /boot/initrd.img-3.13.0-35-generic\ndone\ngrub-install: error: /usr/lib/grub/modinfo.sh doesn't exist. Please specify --target or --directory.\nfailed to install grub!\nUnexpected error while running command.\nCommand: ['install-grub', '--uefi', '/tmp/tmpBpbjAX/target', '/dev/fioa']\nExit code: 1\nReason: -\nStdout: ''\nStderr: ''\n"
Stderr: ''
Success
umount: /tmp/tmpBpbjAX/target: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
Unexpected error while running command.
Command: ['umount', '/tmp/tmpBpbjAX/target']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
2014-10-21 13:04:31,654 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [3]
2014-10-21 13:04:31,656 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2014-10-21 13:04:31,656 - util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.5 finished at Tue, 21 Oct 2014 13:04:32 +0000. Datasource DataSourceMAAS [http://10.245.0.10/MAAS/metadata/curtin]. Up 122.36 seconds
"

[Larry Michel (lmic)]

Adding the log files.

[Blake Rouse (blake-rouse)]

This is an issue with curtin, not MAAS.

Have you had this issue multiple times, is it repeatable? Just want to rule out any hardware issues first.

[Larry Michel (lmic)]

Yes it is repeatable. The logs are from a recreate.

[Blake Rouse (blake-rouse)]

What stream are you getting your images from in MAAS, daily or release?

[Larry Michel (lmic)]

We've recently started using daily (this week). We were previously using release.

[Blake Rouse (blake-rouse)]

Hm. Wonder if that has something to do with it. Looks like a file is missing for the grub installation.

[Larry Michel (lmic)]

Do we have a fix for this issue?

[Blake Rouse (blake-rouse)]

You have change the summary does this issue happen on all machines deploying with UEFI? Or just ones with fioa devices?

[Larry Michel (lmic)]

Yes, seems to be happening for other devices as well. Here's a recreate with latest image:

Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-3.13.0-39-generic
Found initrd image: /boot/initrd.img-3.13.0-39-generic
done
grub-install: error: /usr/lib/grub/modinfo.sh doesn't exist. Please specify --target or --directory.
failed to install grub!
Unexpected error while running command.
Command: ['install-grub', '--uefi', '/tmp/tmpCk8Ljc/target', '/dev/sdc']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [59.7 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [352 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [5820 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [216 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9365 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [159 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [1736 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [108 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [151 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [14 B]\nGet:25 http://archive.ubuntu.com trusty-security/universe amd64 Packages [50.4 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1145 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [74.3 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [14 B]\nGet:30 http://archive.ubuntu.com trusty-security/uni...

[Larry Michel (lmic)]

I didn't realize the stdout was so long. I would have truncated it..

[Andres Rodriguez (andreserl)]

Blake,

Can you please investigate this ASAP. There are other systems that have been affected by UEFI.

Thanks

[Blake Rouse (blake-rouse)]

This seems to be affected in the daily images. I have tested this on my NUC with UEFI and have the same issue.

I want to have smoser to look at this, to see if its a image issue or something we need to do in curtin.

[Rod Smith (rodsmith)]

FWIW, I've run into this, too, with MAAS 1.7 and UEFI. The first error I noticed was that /usr/lib/grub/modinfo.sh doesn't exist. It DOES exist one further directory down on the mounted filesystem (/tmp/tmpqRoeEj/target/usr/lib/grub/x86_64-efi/modinfo.sh). I'm not sure if the installer is trying to read from the installed system or from the tempfs at this point, but adding that one directory element to the script that calls modinfo.sh might be all that's needed to fix the bug.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr196-0ubuntu1

---------------
curtin (0.1.0~bzr196-0ubuntu1) vivid; urgency=medium

  * New upstream snapshot.
    * fix bug installing to UEFI systems (LP: #1383727)
 -- Scott Moser <email address hidden> Mon, 08 Dec 2014 20:01:44 -0500

[Rod Smith (rodsmith)]

This is an extremely critical bug for installations with EFI-mode systems. When will the fix work its way into earlier distributions?

[Jeff Lane (bladernr)]

Just a follow up, when can we expect to see this resolved in Trusty as we move users/testers over to MAAS 1.7.1 in Trusty?

[Chris J Arges (arges)]

Hello Larry, or anyone else affected,

Accepted curtin into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr196-0ubuntu1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Larry, or anyone else affected,

Accepted curtin into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr196-0ubuntu1~14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andres Rodriguez (andreserl)]

Hi Chris,

This has been tested both in Utopic and Trusty and is proven to work as expected. Marking verification-done.

Thanks.

[Rod Smith (rodsmith)]

We're still having problems with one MAAS server. We suspect an ephemeral isn't being updated correctly. What's the best procedure to ensure that the fix gets onto a working MAAS server without wiping out the existing node list?

[Blake Rouse (blake-rouse)]

curtin is not embedded in to the ephemeral, it is copied over user-data.

So if you have the new package a restart to apache2 is all you need.

[Andres Rodriguez (andreserl)]

@Roderick,

We have tested this in OIL and other servers and this works as expected. If you are having issues might be related to your specific hardware. However, as Blake mentioned, you need to restart apache2 if you upgrade curtin.

[Rod Smith (rodsmith)]

It was the curtin-common package at fault. I had been under the mistaken impression that it was the ephemerals that needed to be updated, and we hadn't updated the packages on the server in a while. Once we did, it started working.

[Brian Murray (brian-murray)]

@Roderick - so it was an issue with your upgrade process and not with the SRU itself - is that correct?

[Rod Smith (rodsmith)]

Yes, Brian; AFAIK the SRU is fine.

[Scott Kitterman (kitterman)]

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr196-0ubuntu1~14.04.1

---------------
curtin (0.1.0~bzr196-0ubuntu1~14.04.1) trusty-proposed; urgency=medium

  * New upstream snapshot.
    * fix bug installing to UEFI systems (LP: #1383727)
 -- Andres Rodriguez <email address hidden> Mon, 16 Feb 2015 15:05:57 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr196-0ubuntu1~14.10.1

---------------
curtin (0.1.0~bzr196-0ubuntu1~14.10.1) utopic-proposed; urgency=medium

  * New upstream snapshot.
    * fix bug installing to UEFI systems (LP: #1383727)
 -- Scott Moser <email address hidden> Mon, 08 Dec 2014 20:01:44 -0500

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.