unsupported configuration: machine type 's390-ccw-virtio-mantic' does not support ACPI

[Felipe Reyes]

[Impact]

Fails to launch instances on s390x. The following error is raised:

Details: Fault: {'code': 500, 'created': '2023-11-19T18:28:12Z', 'message': "Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 66eebf5b-01fa-48b8-a8de-58dcf9787fd0. Last exception: unsupported configuration: machine type 's390-ccw-virtio-mantic' does not support ACPI"}. Request ID of server operation performed before checking the server status req-4d52df6d-f4de-486e-8831-08757d083ef9.

[Environment]

Ubuntu 23.10
OpenStack 2023.2 (bobcat)
Nova 3:28.0.0-0ubuntu1
libvirt 9.6.0-1ubuntu1
qemu-system-s390x 1:8.0.4+dfsg-1ubuntu3

[Test Case]

Launch an instance on a s390x hypervisor

Expected result: the new instance reaches ACTIVE state

Actual result: the instance fails to start with the following error: "unsupported configuration: machine type 's390-ccw-virtio-mantic' does not support ACPI"

[Felipe Reyes (freyes)]

I was taking a look to "virsh capabilities" output (see below) on s390x, which doesn't advertise acpi support, this is correct.

# virsh capabilities
<capabilities>

  <host>
    <uuid>94c32186-79cc-471e-909a-9fd63cc039c2</uuid>
    <cpu>
      <arch>s390x</arch>
      <topology sockets='1' dies='1' cores='8' threads='1'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='1024'/>
    </cpu>
    <power_management/>
    <iommu support='no'/>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>20321928</memory>
          <pages unit='KiB' size='4'>5080482</pages>
          <pages unit='KiB' size='1024'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='2' die_id='0' core_id='0' siblings='0-1'/>
            <cpu id='1' socket_id='2' die_id='0' core_id='0' siblings='0-1'/>
            <cpu id='2' socket_id='2' die_id='0' core_id='1' siblings='2-3'/>
            <cpu id='3' socket_id='2' die_id='0' core_id='1' siblings='2-3'/>
            <cpu id='4' socket_id='3' die_id='0' core_id='2' siblings='4-5'/>
            <cpu id='5' socket_id='3' die_id='0' core_id='2' siblings='4-5'/>
            <cpu id='6' socket_id='3' die_id='0' core_id='3' siblings='6-7'/>
            <cpu id='7' socket_id='3' die_id='0' core_id='3' siblings='6-7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>apparmor</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+64055:+108</baselabel>
      <baselabel type='qemu'>+64055:+108</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='s390x'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-s390x</emulator>
      <machine maxCpus='248'>s390-ccw-virtio-mantic</machine>
      <machine canonical='s390-ccw-virtio-mantic' maxCpus='248'>s390-ccw-virtio</machine>
      <machine maxCpus='248'>s390-ccw-virtio-7.2</machine>
      <machine maxCpus='248'>s390-ccw-virtio-kinetic</machine>
      <machine maxCpus='248'>s390-ccw-virtio-4.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-artful</machine>
      <machine maxCpus='248'>s390-ccw-virtio-5.2</machine>
      <machine maxCpus='248'>s390-ccw-virtio-3.1</machine>
      <machine maxCpus='248'>s390-ccw-virtio-groovy</machine>
      <machine maxCpus='248'>s390-ccw-virtio-hirsute</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.12</machine>
      <machine maxCpus='248'>s390-ccw-virtio-disco</machine>
      <machine maxCpus='248'>s390-ccw-virtio-yakkety</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.6</machine>
      <machine maxCpus='248'>s390-ccw-virtio-eoan</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.9</machine>
      <machine maxCpus='248'>s390-ccw-virtio-8.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-6.0</machine>...

[Felipe Reyes (freyes)]

The same version of nova on jammy (via Cloud Archive) doesn't have this problem, so the root cause should be in the interactions between nova and a newer libvirt/qemu or the kernel version.

[Felipe Reyes (freyes)]

Changing the machine type to s390-ccw-virtio-jammy produces same error

Last exception: unsupported
configuration: machine type \'s390-ccw-virtio-jammy\' does not support ACPI\n

[Felipe Reyes (freyes)]

The domain xml generated by Nova includes acpi and apic , that's what's making things fail

```
  <features>
    <acpi/>
    <apic/>
  </features>
```

See attached file to get the full xml definition

[Felipe Reyes (freyes)]

cpaelzer had the chance to test launching (manually) a libvirt based instance on jammy and mantic
passing the <acpi /> section, and he could confirm that on jammy the vm runs while on mantic doesn't
, on jammy the section is ignored, so effectively there is no change in behavior other than libvirt
on mantic is stricter on the xml validation.

[Felipe Reyes (freyes)]

This is the commit that made the <acpi /> node fail - https://github.com/libvirt/libvirt/commit/d8e95ab6b7c45acc121746e2e24edcfca3d8d8a0

[Felipe Reyes (freyes)]

I was taking a look to nova's source code to determine what would be a good approach to fix this issue, so far what I've found is:

- Nova always add `<acpi />` when the instance is qemu or kvm[0]
- At the moment the <features></features> tree is not being parsed when reading the capabilities supported by the host[1]

So a fix would involve creating a new class named "LibvirtConfigCapsFeatures" that implements a `parse_dom()` method to read, in this first iteration, the acpi node.

[0] https://github.com/openstack/nova/blob/cd5b7e806855fa37461bc2da865435b16408fbd3/nova/virt/libvirt/driver.py#L6203
[1] https://github.com/openstack/nova/blob/cd5b7e806855fa37461bc2da865435b16408fbd3/nova/virt/libvirt/config.py#L120C1-L131C1