[iotg] Desktop Image doesn't boot if additional USB Disks are present

[Sachin Mokashi]

The image released for iotg[1] is a preinstalled disk image. After flashing it on a USB and booting from it, the Image doesn't boot if any additional USB disks are present.

This has been reproduced on Intel EHL CRB and by AAEON team.

Workaround: The user needs to disconnect any other USB drives present before booting it from the Ubuntu USB

[1] https://cdimage.ubuntu.com/releases/focal/release/inteliotg/ubuntu-20.04-preinstalled-desktop-amd64+intel-iot.img.xz

[Doug Jacobs (djacobs98)]

Could this be because of the boot order on the specific device?

With the Intel BIOS, you can bring up a boot menu, and choose the specific device you want to boot immediately. In this situation, multiple USB drives are not a problem, as long as the user knows which one is which. This is method I prefer, because I always know which device I am booting.

But some other BIOS, you can only change the boot order. For this situation, it's best you only have the USB drive with the live image connected, otherwise the board might try to boot the other drive(s). Sometimes this might do the right thing and try each USB drive in succession, until one boots. However, I've found this behavior to be unpredictable, especially since other options like PXE Boot can also get called, and if PXE Boot fails, it does not cause the BIOS to try the next entry in the boot table.

For this reason, I always boot with just the live USB drive inserted. Once Ubuntu has booted, then I insert the data drive to reimage the local storage device.

I did look at a way to add the image file to the drive with the live bootable image, so you would not need 2 separate USB drives however the procedure is a bit complicated and requires you to have Ubuntu installed on a separate system. So that is why I have stuck with the 2-USB-Drive method.

[ethan.hsieh (ethan.hsieh)]

When an additional USB Disk is not present:
grub> ls
(proc) (hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1)

When an additional USB Disks is present:
grub> ls
(proc) (hd0) (hd0,gpt1) (hd1) (hd1,gpt3) (hd1,gpt2) (hd1,gpt1)

When an additional USB Disks is present, the sequence of disks changes.
But, the grub config still sets 'hd0,gpt3' as root.

EFI/ubuntu/grub.cfg:
---
            set root='hd0,gpt3'
            search --label $label --set=writable
            echo 'Loading Linux ...'
                linux /boot/vmlinuz root=LABEL=$label ro console=tty1 console=ttyS0 systemd.log_level=debug systemd.debug-shell=1
---

Have to update grub.cfg to fix the issue.

[ethan.hsieh (ethan.hsieh)]

@Sachin

As you mentioned in the bug description,
"The image released for iotg[1] is a preinstalled disk image. After flashing it on a USB and booting from it, the Image doesn't boot if any additional USB disks are present."

> After flashing it on a USB and booting from it
=> The image should be flashed into the internal disk on the machine and the system should boot from the internal disk.
=> Is it typo?

I can see the same error message, but my steps are
1. Insert a bootable Live Ubuntu USB stick
2. Power on the device and then boot to Live ubuntu
3. dd the IoT image to the internal disk
4. Power off the device.
5. Unplug the bootable Live Ubuntu USB stick
6. Insert an empty USB stick (There is only one external USB stick plugged)
7. Power on the device.

[Sachin Mokashi (sachinmokashi)]

@ethan

According to the Instructions[1] shared in the Download Page[2], user can boot the Image from a USB drive. This is helpful incase the platform doesn't have an internal disk.

In any case, the issue occurs if an additional USB disk is present at the time of booting. As you mentioned in #3, it is because the root is still set to 'hd0,gpt3' even though the sequence changes. I believe the root should be set based on the UUID of the device from which it boots, something like: root=UUID=<uuid_of_boot_device>

[1] https://assets.ubuntu.com/v1/590a61f7-Installation+Instructions+Desktop+Image+Intel+IoTG+Platforms.pdf
[2] https://ubuntu.com/download/iot/intel-iotg

[ethan.hsieh (ethan.hsieh)]

@Sachin

According to the instructions[1][2], users have to create a bootable USB drive for flashing image. The bootable USB drive is based on Ubuntu 20.04.3 LTS[3], not the released IoT image.

If You dd the IoT Image to an USB drive and boot from this USB drive, it's bootable.
But, it's not the user behavior we expect.

As I mentioned in comment#3, it's a bug. Canonical will fix it.

We don't set "root" to UUID because this image is a pre-installed image.
But, the users can update the value to UUID by update-grub.

---
[1] https://assets.ubuntu.com/v1/590a61f7-Installation+Instructions+Desktop+Image+Intel+IoTG+Platforms.pdf
[2] https://ubuntu.com/download/iot/intel-iotg
[3] https://releases.ubuntu.com/20.04/

[ethan.hsieh (ethan.hsieh)]

@Brian

There are two issues here. Please help to take a look at these two issues.

1. Please see comment#3
2. grub.cfg in the EFI partition won't get updated when we execute update-grub.
In the IoT image, the real grub.cfg is located at the EFI partition.

In stock Ubuntu, the grub.cfg in the EFI partition is
---
# cat /boot/efi/EFI/ubuntu/grub.cfg
search.fs_uuid XXXXXXXXX root hd0,gpt2
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg
---
The real grub.cfg (/boot/grub/grub.cfg) is in the writable (rootfs) partition.
And, this one is the one which will get updated by update-grub.

[Łukasz Zemczak (sil2100)]

I think the short-term solution would be to update the grub-cpc.cfg file in the classic amd64 gadget to properly handle searching of boot devices (we could probably do that with some `search (...)` combination?) and then re-spinning the intel-IOT images so that they work again in such cases. However updating existing devices is something we still need to figure out how to do, as we basically inherited the old pre-new-world design from older Ubuntu Core images where the boot partition assets are not really getting updated - mostly because preinstalled amd64 images are quite a new thing, with Intel IOT being basically the first official product we've built using this framework. This is something we need to fix very soon, maybe thinking about a better design for it.

[Łukasz Zemczak (sil2100)]

I have some branches ready that should solve this problem + the grub.cfg upgrade scenario that Ethan mentioned (thanks for catching it!). It's still in review phase, I'll be creating some test images and if everything looks good, we can think of fast-tracking it somehow.

Basically these two changes need to land:
https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+merge/410949
https://github.com/snapcore/pc-amd64-gadget/pull/54

[Pierre Equoy (pieq)]

HW used: Aaeon EHL board
CID: 202109-29496
Image used: https://cdimage.ubuntu.com/focal/daily-preinstalled/20211102.1/ (with -proposed enabled)

After installation completed, I inserted a USB drive with some files on it, and rebooted the device. It didn't hang anymore, and booted from the correct partition (/dev/mmcblk0p3).

[Doug Jacobs (djacobs98)]

Image: https://cdimage.ubuntu.com/focal/daily-preinstalled/20211102.1/
HW: Aaeon TGL board

Installed the image, verified with Disks utility that the data partition was properly expanded.

Rebooted the device, leaving my data drive plugged in, and the desktop booted as expected.

[Pierre Equoy (pieq)]

HW used: Aaeon EHL board
CID: 202109-29496
Image used: http://cdimage.ubuntu.com/focal/daily-preinstalled/20211108.1/ (with -proposed DISABLED)

After installation completed, I inserted a USB drive with some files on it, and rebooted the device. It didn't hang anymore, and booted from the correct partition (/dev/mmcblk0p3).

→ OK

[Pierre Equoy (pieq)]

HW used: Aaeon EHL board
CID: 202109-29496
Image used: http://cdimage.ubuntu.com/focal/daily-live/20211202.2/focal-desktop-amd64+intel-iot.iso (with -proposed DISABLED)

After installation completed, I inserted a USB drive with some files on it, and rebooted the device. It didn't hang anymore, and booted from the correct partition.

→ OK

[Sachin Mokashi (sachinmokashi)]

@rebeccasoffe, As mentioned by Pierre this bug is not to be reproduced anymore, can we Close this issue?