Calamares fails in UEFI mode after external command in Lubuntu Cosmic

[sudodus]

Error

Installation Failed

External command failed to finish.
Command [ -d /sys/firmware/efi ] && chroot /tmp/calamares-root-ef72qjas apt-cdrom add && chroot /tmp/calamares-root-ef72qjas sed -e '/deb http/ s/^#*/#/' -i /etc/apt/sources.list && chroot /tmp/calamares-root-ef72qjas apt update && chroot /tmp/calamares-root-ef72qjas apt -y install grub-efi-amd64-bin && chroot /tmp/calamares-root-ef72qjas sed -i 's/#deb/deb/' /etc/apt/sources.list || exit 0 failed to finish in 60 seconds. There was no output from the command.

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: calamares 3.2.1-1
ProcVersionSignature: Ubuntu 4.15.0-23.25-generic 4.15.18
Uname: Linux 4.15.0-23-generic x86_64
ApportVersion: 2.20.10-0ubuntu5
Architecture: amd64
CasperVersion: 1.394
CurrentDesktop: LXQt
Date: Tue Jul 10 12:56:59 2018
LiveMediaBuild: Lubuntu 18.10 "Cosmic Cuttlefish" - Alpha amd64 (20180710)
SourcePackage: calamares
UpgradeStatus: No upgrade log present (probably fresh install)

[sudodus (nio-wiklund)]

I tested in a Toshiba laptop from 2013 with Intel i5 (generation 3),

http://www.toshiba.se/laptops/satellite-pro/c850/satellite-pro-c850-19w/

and using the default location (New York and US English)

- in UEFI mode (not secure boot)

- 'Erase disk' (to check with an MSDOS partition table, if Calamares can create an EFI system partition etc). It 'says' that it will create a GPT, an ESP and a root partition ...

Indeed it did that:

---
lubuntu@lubuntu:~$ sudo parted /dev/sda p
Model: ATA KINGSTON SUV400S (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 2098kB 317MB 315MB fat32 boot, esp
 2 317MB 120GB 120GB ext4

lubuntu@lubuntu:~$
---

But when the installer was at 78%, it did not get any further, and the error window appeared.

[Simon Quigley (tsimonq2)]

https://phab.lubuntu.me/rCALASETTINGS756ccfef18776e8a73893dfe34513408712702b5

[Launchpad Janitor (janitor)]

This bug was fixed in the package calamares-settings-ubuntu - 5

---------------
calamares-settings-ubuntu (5) cosmic; urgency=medium

  * Add translations support for the desktop file and the installer slideshow.
  * Increase the timeout for the shellprocess commands (LP: #1781015).
  * Remove the rm line of the shellprocess command (LP: #1780977).

 -- Simon Quigley <email address hidden> Tue, 10 Jul 2018 19:17:43 -0500

[sudodus (nio-wiklund)]

This bug is still alive with the daily iso file dated July 11 2018 :-(

When the installer was at 78%, it did not get any further, and the error window appeared (this time after 300 seconds).

[sudodus (nio-wiklund)]

@Simon,

But I want to add that in BIOS mode (alias CSM in this Toshiba laptop) it is possible to install Lubuntu Cosmic from the cosmic-desktop-amd64.iso file dated dated July 11 2018.

This is a great step forward :-)

So now we can focus on UEFI mode and in the next step secure boot.

[Simon Quigley (tsimonq2)]

Hello Nio, thanks for the testing!

Could you please replace /etc/calamares/modules/shellprocess.conf with the attached file before the install so I can see what exactly is wrong here?

The weird part is, I can't reproduce this.

[sudodus (nio-wiklund)]

@Simon,

Is it enough with this feedback? I started Calamares from the desktop icon and got the following output to an 'error window'.

---
External command failed to finish.
Command [ -d /sys/firmware/efi ] && chroot /tmp/calamares-root-5b8t306g apt-cdrom add || exit 0 failed to finish in 300 seconds. There was no output from the command.
---

[sudodus (nio-wiklund)]

I can add

- There is a working DVD drive in my Toshiba laptop. But it is empty (no disk inserted) during these tests.

- This is still tested with the daily iso file dated July 11 2018 (zsynced yesterday).

- I cloned the iso file to a USB pendrive. There is no problem to run it live in my Toshiba; it works in UEFI mode as well as in BIOS mode. (But Calamares works only in BIOS mode.)

[Simon Quigley (tsimonq2)]

Ahh, that makes sense. The working DVD drive is a dead giveaway.

Playing with potential solutions now.

Thanks!

[Simon Quigley (tsimonq2)]

Actually, since I can't actually reproduce this, one more thing.

Could you try this config file now?

[sudodus (nio-wiklund)]

@Simon,

Before it stopped at 78%, but with this config file, it reached 82%, so '-m' made a difference.

---Boost.Python error in job "bootloader".
Command 'grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=lubuntu --force' returned non-zero exit status 1.
grub-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist. Please specify --target or --directory.

Traceback:
File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 420, in run
    prepare_bootloader(fw_type)

  File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 387, in prepare_bootloader
    install_grub(efi_directory, fw_type)

  File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 261, in install_grub
    "--force"])

  File "<string>", line 4, in <module>
---

[Simon Quigley (tsimonq2)]

Hi Nio,

The attached *should* fix the problem. It would be great if you could try it.

Thanks.

[sudodus (nio-wiklund)]

I'm sorry Simon,

But it fails too :-(

This bug is still alive with the daily iso file dated July 11 2018 :-(

When the installer was at 78%, it did not get any further, and the error window appeared.

---
External command failed to finish.
Command [ -d /sys/firmware/efi ] && chroot /tmp/calamares-root-htc9thyf apt-cdrom add -d=/cdrom/ || exit 0 failed to finish in 300 seconds. There was no output from the command.
---

[sudodus (nio-wiklund)]

Hi again Simon,

I zsynced to get the current daily iso file cosmic-desktop-amd64.iso dated July 12 2018, and tried again in my Toshiba booting in UEFI mode.

This time the system selected Stockholm (no longer New York), and I continued with this default setting (Swedish time and language) and selected 'Erase disk' as before. I also installed your new custom shellprocess.conf before starting Calamares.

It failed in the same way at 78% and with the same message as reported in comment #16 (for the iso file dated July 11 2018).

[Simon Quigley (tsimonq2)]

20180714 should fix this.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[sudodus (nio-wiklund)]

Hi again Simon,

I zsynced to get the current daily iso file cosmic-desktop-amd64.iso dated July 14 2018, and tried again in my Toshiba booting in UEFI mode.

I used the system as it was cloned from the iso file and with default settings as much as possible, and with 'Erase disk'.

It reached 82% and failed with the following error message.

---
Error

Installation failed

Command 'grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=lubuntu --force' returned non-zero exit status 1.
grub-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist. Please specify --target or --directory.

Traceback:
File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 420, in run
    prepare_bootloader(fw_type)

  File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 387, in prepare_bootloader
    install_grub(efi_directory, fw_type)

  File "/usr/lib/x86_64-linux-gnu/calamares/modules/bootloader/main.py", line 261, in install_grub
    "--force"])

  File "<string>", line 4, in <module>
---

[sudodus (nio-wiklund)]

I swapped off 'everything', in this case the zram, and tried again in the Toshiba laptop like in the previous comment. The error [message] was exactly the same, so we should *not* blame zram.

[sudodus (nio-wiklund)]

I tested in the Intel NUC, described in the duplicate bug report

https://bugs.launchpad.net/ubuntu/+source/calamares/+bug/1781540

I swapped off an existing swap partition in the target drive (but not the zram), and tried like in the previous comments (# 20-21).

The error [message] was exactly the same, so we should *not* blame the DVD drive or some unique hardware component or unique UEFI-BIOS system.

@Simon,

Something goes wrong when trying to install the bootloader in UEFI mode and it does not find a file. Have you tried according to the 'built-in tip' to specify --target or --directory?

[Simon Quigley (tsimonq2)]

How about 20180715.1?

The underlying problem is that the CD sources aren't being added correctly to pull in the package. That's why /usr/lib/grub/x86_64-efi/modinfo.sh is missing; because it's not installed.

[sudodus (nio-wiklund)]

Hi again Simon,

I zsynced to get the current daily iso file cosmic-desktop-amd64.iso dated July 15 2018 (20180715.1).

The test in my Toshiba failed:

---
Error

Installation Failed

External command failed to finish.
Command [ -d /sys/firmware/efi ] && chroot /tmp/calamares-root-hvf11ezl apt-cdrom add -d=/media/apt/ && chroot /tmp/calamares-root-hvf11ezl sed -e '/deb http/ s/^#*/#/' -i /etc/apt/sources.list && chroot /tmp/calamares-root-hvf11ezl apt update && chroot /tmp/calamares-root-hvf11ezl apt -y install grub-efi-amd64-bin && chroot /tmp/calamares-root-hvf11ezl sed -i 's/#deb/deb/' /etc/apt/sources.list && sed -i '/deb cdrom/d' /tmp/calamares-root-hvf11ezl/etc/apt/sources.list || exit 0 failed to finish in 300 seconds. There was no output from the command.
---

[sudodus (nio-wiklund)]

... and it is back getting stuck at 78% where it is timing out.

[sudodus (nio-wiklund)]

Installation fails the same way in the NUC.

[Simon Quigley (tsimonq2)]

I'll get my hands on real hardware later this week to test it.

Thanks for your help Nio, and I'll let you know if I can't reproduce it on any of my hardware.

[Simon Quigley (tsimonq2)]

After getting my hands on some real hardware, I was able to properly reproduce this, hunt down the problem, and get a fix.

Uploading nowish, and the fix should be included in 20180719. Later today I'll play with getting Secure Boot working.

Thanks for all your help Nio, it's very much appreciated. :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package calamares-settings-ubuntu - 10

---------------
calamares-settings-ubuntu (10) cosmic; urgency=medium

  * Fix non-secureboot UEFI installs on real hardware (LP: #1781015).

 -- Simon Quigley <email address hidden> Wed, 18 Jul 2018 23:20:42 -0500

[sudodus (nio-wiklund)]

Congratulations and thanks Simon,

I tested in the NUC, and Calamares installed Lubuntu Cosmic in UEFI mode. You have squashed this bug :-)

(I will do some more tests, but I'm sure that you have solved the main problem.)

[Simon Quigley (tsimonq2)]

Please, do more tests :)

[sudodus (nio-wiklund)]

I tested in my Toshiba laptop from 2013 with Intel i5 (generation 3),

http://www.toshiba.se/laptops/satellite-pro/c850/satellite-pro-c850-19w/

I selected Swedish language and Stockholm time (and 'Erase disk' and of course in UEFI mode).

Calamares worked all the way, and the installed system works.

There was only a minor glitch: When rebooting after the installation, the system was stuck at the plymouth '5 dots' until I did 'SysRq REISUB'. This glitch is often said to be caused by a race condition and affects some computers with some versions of operating systems.