FAIL: curtin command curthooks

[Chris Gregan]

MAAS nodes fail to deploy due to:

Jun 15 11:47:38 ralts cloud-init[1656]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
Jun 15 11:47:38 ralts cloud-init[1656]: Traceback (most recent call last):
Jun 15 11:47:38 ralts cloud-init[1656]: File "/curtin/curtin/commands/main.py", line 201, in main
Jun 15 11:47:38 ralts cloud-init[1656]: ret = args.func(args)
Jun 15 11:47:38 ralts cloud-init[1656]: File "/curtin/curtin/commands/curthooks.py", line 1103, in curthooks
Jun 15 11:47:38 ralts cloud-init[1656]: setup_grub(cfg, target)
Jun 15 11:47:38 ralts cloud-init[1656]: File "/curtin/curtin/commands/curthooks.py", line 388, in setup_grub
Jun 15 11:47:38 ralts cloud-init[1656]: uefi_remove_old_loaders(grubcfg, target)
Jun 15 11:47:38 ralts cloud-init[1656]: File "/curtin/curtin/commands/curthooks.py", line 236, in uefi_remove_old_loaders
Jun 15 11:47:38 ralts cloud-init[1656]: ['efibootmgr', '-B', '-b', entry], capture=True)
Jun 15 11:47:38 ralts cloud-init[1656]: File "/curtin/curtin/util.py", line 671, in subp
Jun 15 11:47:39 ralts cloud-init[1656]: return subp(*args, **kwargs)
Jun 15 11:47:39 ralts cloud-init[1656]: File "/curtin/curtin/util.py", line 263, in subp
Jun 15 11:47:39 ralts cloud-init[1656]: return _subp(*args, **kwargs)
Jun 15 11:47:39 ralts cloud-init[1656]: File "/curtin/curtin/util.py", line 131, in _subp
Jun 15 11:47:39 ralts cloud-init[1656]: cmd=args)
Jun 15 11:47:39 ralts cloud-init[1656]: curtin.util.ProcessExecutionError: Unexpected error while running command.
Jun 15 11:47:39 ralts cloud-init[1656]: Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpnwei2asf/target', 'efibootmgr', '-B', '-b', '0011']
Jun 15 11:47:39 ralts cloud-init[1656]: Exit code: 15
Jun 15 11:47:39 ralts cloud-init[1656]: Reason: -
Jun 15 11:47:39 ralts cloud-init[1656]: Stdout: ''
Jun 15 11:47:39 ralts cloud-init[1656]: Stderr: Could not delete variable: Input/output error
Jun 15 11:47:39 ralts cloud-init[1656]:
Jun 15 11:47:39 ralts cloud-init[1656]: Unexpected error while running command.
Jun 15 11:47:39 ralts cloud-init[1656]: Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpnwei2asf/target', 'efibootmgr', '-B', '-b', '0011']

[Ryan Harper (raharper)]

Would you be able to post (or at least point to which file in that
infra tarball that contains the file from which this snippet is
posted)?

On Mon, Jun 18, 2018 at 3:25 PM Chris Gregan <email address hidden> wrote:
>
> ** Attachment added: "infra-logs.tar"
> https://bugs.launchpad.net/curtin/+bug/1777510/+attachment/5154061/+files/infra-logs.tar
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1777510
>
> Title:
> FAIL: curtin command curthooks
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1777510/+subscriptions

[Ryan Harper (raharper)]

The failure is when curtin attempts to clean up "old" efi menu entries. We don't have enough debug in there to see what it found in total but it appears that one of the parsed entries '0011' does not exist.

It would be nice to see a dump of:

efibootmgr -v

Then curtin matches, r'^.*File\(\\EFI.*$'
and for each of those matches will call:

'efibootmgr', '-B', '-b', entry

[Jason Hobbs (jason-hobbs)]

That would seem to fall under bug 1607345 - shouldn't curtin be gathering the debug info it needs?

[Jason Hobbs (jason-hobbs)]

Could we modify our curtin preseed to collect this? When does it need to run?

[Ryan Harper (raharper)]

Well, there aren't curtin error-logs here, but even if they were, as
you say, we need to add grub/efi config dumping.

That said, even if we saw efibootmgr -v and it showed an entry with
0011 present, which triggers the path, curtin
doesn't know why efibootmgr fails when attempting to remove it.

That's more platform/EFI/grub issue.

The efi entries are stored in the bios, so it won't matter when you
query them. You may also find the info if you enter the bios itself
to see what entries it currently has configured.

If I had to guess, it's likely secure-boot related.

Does this also fail deploying Xenial or Bionic only?
On Mon, Jun 18, 2018 at 4:25 PM Jason Hobbs <email address hidden> wrote:
>
> Could we modify our curtin preseed to collect this? When does it need to
> run?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1777510
>
> Title:
> FAIL: curtin command curthooks
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1777510/+subscriptions

[Launchpad Janitor (janitor)]

[Expired for curtin because there has been no activity for 60 days.]