[SRU] Boot from ISO does not work

[Pavlo Shchelokovskyy]

[Impact]

Pure ISO image cannot be booted even though CVEs(https://review.opendev.org/c/openstack/ossa/+/923301/1/ossa/OSSA-2024-001.yaml) are already supported.

[Where problems could occur]

ISO+MBR/GPT multiple images can be booted in disk mode with CVE detection support. However, a pure ISO single image can only be booted in cdrom format. cdrom mode remains unsupported without this fix even if CVEs are already supported since Bobcat release.

[Test Case]

Pls refer to [Test steps] section below.

[Regression Potential]

There are two patches, one is deepcopy patch(https://review.opendev.org/c/openstack/nova/+/920374), introduced since 30.0.0;
the other one is iso patch(https://review.opendev.org/c/openstack/nova/+/909611), introduced since 31.0.0
The fixes are already in the upstream main, epoxy(2025.1), need to backport to dalmatian(2024.2), caracal(2024.1), bobcat(2023.2)

I have tested this fix, it worked fine - https://bugs.launchpad.net/cloud-archive/+bug/2054446/comments/21

[Others]

Original Bug Description Below
===========

It may be https://bugs.launchpad.net/nova/+bug/1454901 resurfacing again..

Symptoms using fresh DevStack/master:

I follow the docs https://docs.openstack.org/nova/latest/user/launch-instance-using-ISO-image.html
and using tinycore iso for testing http://tinycorelinux.net/ (this is very small liveCD ISO) to speed up testing.

Image is created with

openstack image create --public --file Core-14.0.iso --disk-format iso Core-14.0.iso

Then I boot the instance as usual

openstack --os-compute-api-version 2.latest server create --image Core-14.0.iso --flavor cirros256 --no-network iso-test

The instance is ACTIVE, but when I connect to it via noVNC it shows that it failed to boot - "No bootable device"

The relevant part of the instance XML is

  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/opt/stack/data/nova/instances/0c3d31a9-7ecf-4625-9e0e-4cd89d1b76c2/disk' index='1'/>
      <backingStore type='file' index='2'>
        <format type='raw'/>
        <source file='/opt/stack/data/nova/instances/_base/b2b7eb374cc75a24d0e5ba0eca7ca9cc1e6dc7c6'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>

This is Nova/master + libvirt 8.0

I checked the same on OpenStack Antelope - the result is the same.

However, on OpenStack Queens (+ libvirt 4.0) the instance boots from the same ISO image uploaded to Glance just fine! The relevant part of libvirt domain XML in OpenStack Queens is

  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/791cf357-02e6-4a7f-9310-25f5e79cf27d/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/d646a5bfc2ce7e3926d0f368b8adb975b245cfd2'/>
        <backingStore/>
      </backingStore>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

Notice the difference in disk device and target/address/alias. When I manually edited the XML on devstack to look like that from Queens, the instance booted successfully.

This looks like a regression somewhere in Nova (libvirt driver?)

[Test steps]

Pls refer to the comment (https://bugs.launchpad.net/cloud-archive/+bug/2054446/comments/21) for specific test steps.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/909611

[Pavlo Shchelokovskyy (pshchelo)]

Yoga also works fine.

It seems the culprit is this change
https://review.opendev.org/c/openstack/nova/+/826527/8/nova/virt/libvirt/blockinfo.py#417

this is the only place where 'iso' disk_format is checked.
Previously this branch of code was run on any 'local image', now it does not, thus ISO images are treated like normal disks.

[Alejandro Garcia (agarciaws)]

We have faced this issue today in our environment after a recent upgrade to Zed. Any workaround we could try for this?

[Joris Vleminckx (jvleminc)]

What a coincidence, we stumbled upon this also today... our customer is bit stuck now.

[Alejandro Garcia (agarciaws)]

FYI, we have been able to workaround this by setting the hw_firmware_type property to uefi on the image:

https://docs.openstack.org/nova/zed/admin/uefi.html

[sean mooney (sean-k-mooney)]

by the way we were dealing with cves at the time so i didn't say this a few weeks ago.

booting form iso does still work but only in a subset of cases

it works if the iso file is created with a bootable filesystem i.e. if the iso is built so that it will also work on a USB.

tinycore 14 does not work but 15 does.
as does the apline isos.

that is why this has not been fixed for so long.

modern isos work. onces that are designed to only work if burned to a cd, i.e. without ISOLINUX and a boot loader in the system area of the the iso don't.

[Carlos Bravo (carlosbravo)]

any iso that is windows based (windows installers or veeam server recovery for windows for instance) doesn't boot. Any way I can inject any type of preboot to get them to work? This is a big stopper for us at the moment.

[Pavlo Shchelokovskyy (pshchelo)]

Given recent additions regarding image format introspection, now it seems no ISO can be booted, as the 'iso with bootable filesystem' Sean mentions now fail to start with

Image ... is unacceptable: Image content does not match disk_format
which in turn is caused by
oslo_utils.imageutils.format_inspector.ImageFormatError: Multiple formats detected: gpt,iso

[Pavlo Shchelokovskyy (pshchelo)]

So, to summarize:

currently nova assigns 'disk' device type to ISO image in the libvirt XML.

This does not work for pure ISO images (many out there, I presume Windows is definitely one of those, and older *nix systems too), instance becomes ACTIVE but does not actually boot due to 'no bootable device'.

This should work for ISOs that also have ISOLINUX and bootloader too (newer *nix images, tested with latest ubuntu-server-24.04 and alpine), but this is now blocked by image format introspection.

[Hua Zhang (zhhuabj)]

I did some tests using jammy-bobcat, but hit the error 'Image not in a supported format', pls see the paste[0] for more details.

I suspected that it might be due to jammy-bobcat not including this cve fix[1], so then I set up a devstack env with the master branch to do some more tests, but the results were the same. Even without applying the iso patch[3], both tests with disk-format=raw and disk-format=iso resulted in the error 'Image content does not match' disk_format, pls see the paste[2] for more details.

[0] https://paste.ubuntu.com/p/dvpgpb6mGn/
[1] https://review.opendev.org/c/openstack/nova/+/923729/1/nova/virt/images.py
[2] https://paste.ubuntu.com/p/VKtM8Nb2qT/
[3] https://review.opendev.org/c/openstack/nova/+/909611

[Dmitriy Chubinidze (chubinidzedr)]

In Dalmatian the same problem with rescue from qcow2 format, but in that case server falls in error state:

2024-12-18 15:43:09.299 4259 ERROR oslo_messaging.rpc.server [None req-4c642563-3852-4439-951d-dc6cd8785713 39ef1840d2d14de49ba16ac7e11ec2c4 b6f178e22d304f99a905c31f0e374587 - - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 6b4aecc6-7231-4584-970c-53c3bf47cd8f cannot be rescued: Driver Error: Multiple formats detected: gpt,iso

Original rescuecd(6.0.7) was formatted from iso to qcow2.

OS: AlmaLinux 9.5

[Ivan Anfimov (anfimovir)]

Fix available in:
https://review.opendev.org/c/openstack/nova/+/931833
https://review.opendev.org/c/openstack/oslo.utils/+/931695

[Hua Zhang (zhhuabj)]

I have tested this iso patch (https://review.opendev.org/c/openstack/nova/+/909611) with iso_gpt patch (https://review.opendev.org/c/openstack/nova/+/931833), it works well. For more details, pls refer to this paste - https://paste.openstack.org/show/b8TUfJVzOhHY5eaowOYk/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/909611
Committed: https://opendev.org/openstack/nova/commit/96a5c21f24f6a89411d12677ae5fa75adf26ed61
Submitter: "Zuul (22348)"
Branch: master

commit 96a5c21f24f6a89411d12677ae5fa75adf26ed61
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Feb 20 19:47:44 2024 +0000

    Fix device type when booting from ISO image

    Closes-Bug: #2054446
    Change-Id: Ib4cc34c30a8acf09a1ab6be89d0faa76fb6e705e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/nova/+/943924

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 31.0.0.0rc1 Epoxy release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/945816

[Yaguang Tang (heut2008)]

To summarize again for affected releases,

The oslo.utils which detect ISO with gpt,iso multi formats issue only affects 2024.2 (2025.1 has been fixed and released), as before 2024.2 oslo.uitls has no image format inspect functionality, and also the format inspect code in Nova has no gpt inspector, so it won't able to detect multi formats for ISO.

To fix release of 2024.2, we need both this fix and the oslo.utils fix to be backport to the 2024.2 release, and Nova need to bump requirements for oslo.utils.

For Nova before 2024.2 relases, only this https://review.opendev.org/c/openstack/nova/+/909611 need to be backported. and it affects only those pure ISO images like windows.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/945903

[Hua Zhang (zhhuabj)]

I tested caracal backport for this patch today, it worked fine - https://paste.ubuntu.com/p/zmsbXf3fn5/

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Yaguang Tang <email address hidden>" on branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/nova/+/945816

[Sam Morrison (sorrison)]

We've just upgraded from yoga to zed and experiencing this bug, wonder if anyone found a fix for zed?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/943924
Committed: https://opendev.org/openstack/nova/commit/a8de6737d22d19fea3d93024fb0cbbe40d67dde9
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit a8de6737d22d19fea3d93024fb0cbbe40d67dde9
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Feb 20 19:47:44 2024 +0000

    Fix device type when booting from ISO image

    Closes-Bug: #2054446
    Change-Id: Ib4cc34c30a8acf09a1ab6be89d0faa76fb6e705e
    (cherry picked from commit 96a5c21f24f6a89411d12677ae5fa75adf26ed61)

[Yaguang Tang (heut2008)]

@sorrison Fix for zed is on the way, we have to backport the fix one by one release, now working on stable/2024.1

[Sam Morrison (sorrison)]

@heut2008 great! We can help test etc. if you need, thanks

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/945816
Committed: https://opendev.org/openstack/nova/commit/b8fcc5fd224cb01728a524594bd08b347f59f7a0
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit b8fcc5fd224cb01728a524594bd08b347f59f7a0
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Feb 20 19:47:44 2024 +0000

    Fix device type when booting from ISO image

    Depends-On: https://review.opendev.org/c/openstack/cinder/+/946101

    Closes-Bug: #2054446
    Change-Id: Ib4cc34c30a8acf09a1ab6be89d0faa76fb6e705e
    (cherry picked from commit 96a5c21f24f6a89411d12677ae5fa75adf26ed61)
    (cherry picked from commit a8de6737d22d19fea3d93024fb0cbbe40d67dde9)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/nova/+/947296

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/945903
Committed: https://opendev.org/openstack/nova/commit/2c0463d70e712925254b1e348927633c8c884b5c
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 2c0463d70e712925254b1e348927633c8c884b5c
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Feb 20 19:47:44 2024 +0000

    Fix device type when booting from ISO image

    Closes-Bug: #2054446
    Change-Id: Ib4cc34c30a8acf09a1ab6be89d0faa76fb6e705e
    (cherry picked from commit 96a5c21f24f6a89411d12677ae5fa75adf26ed61)
    (cherry picked from commit a8de6737d22d19fea3d93024fb0cbbe40d67dde9)
    (cherry picked from commit b8fcc5fd224cb01728a524594bd08b347f59f7a0)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/947431

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 28.3.1 Bobcat release.