Instance creation fails when [libvirt] cpu_mode is custom and its flavor contains hw:mem_encryption

[Takashi Kajinami]

Description
===========
When a user tries to launch an instance with a flavor containing hw:mem_encryption property, the instance always becomes error state if the nova-compute has [libvirt] cpu_mode = custom.
Instance creation succeeds if
- hw_mem_encryption image property is used instead of the flavor property
- [libvirt] cpu_mode is None

Steps to reproduce
==================
1. Set the following options in nova.conf and restart nova-compute

[libvirt]
cpu_mode = custom
cpu_models = EPYC

2. Prepare a flavor with memory encryption enabled

$ openstack flavor show m1.small-enc -f yaml
OS-FLV-DISABLED:disabled: false
OS-FLV-EXT-DATA:ephemeral: 0
access_project_ids: null
description: null
disk: 20
id: ee97652f-8948-4cdd-a5cd-71411cf9c8e4
name: m1.small-enc
os-flavor-access:is_public: true
properties:
  hw:mem_encryption: 'true'
ram: 2048
rxtx_factor: 1.0
swap: 0
vcpus: 1

3. Create an image with hw_firmware_type property set to 'uefi'

$ openstack image show cirros-uefi -f yaml
checksum: c8fc807773e5354afe61636071771906
container_format: bare
created_at: '2023-10-25T02:46:57Z'
disk_format: qcow2
file: /v2/images/d6353363-f580-464c-9909-93212298a58a/file
id: d6353363-f580-464c-9909-93212298a58a
min_disk: 0
min_ram: 0
name: cirros-uefi
owner: 5a2803c4cdb1412fa1e83738d7821904
properties:
  hw_disk_bus: scsi
  hw_firmware_type: uefi
  hw_scsi_model: virtio-scsi
  os_hash_algo: sha512
  os_hash_value: 1103b92ce8ad966e41235a4de260deb791ff571670c0342666c8582fbb9caefe6af07ebb11d34f44f8414b609b29c1bdf1d72ffa6faa39c88e8721d09847952b
  os_hidden: false
  owner_specified.openstack.md5: ''
  owner_specified.openstack.object: images/cirros-uefi
  owner_specified.openstack.sha256: ''
  stores: fs
protected: false
schema: /v2/schemas/image
size: 21430272
status: active
tags: []
updated_at: '2023-10-25T06:00:15Z'
virtual_size: 117440512
visibility: public

4. launch an instance using the flavr and the image
$ openstack server create --image cirros-uefi --flavor m1.small-enc --network private cirros-enc

Expected result
===============
The instance becomes active state

Actual result
=============
Instance becomes error state. The following traceback is found in nova-compute.log

```
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [None req-104288bc-7bf5-4bcd-a728-cd85ac72416f 69d6ccfef7e240398970c80f0be8ccf7 5a2803c4cdb1412fa1e83738d7821904 - - default default] [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] Failed to build and run instance: nova.exception.FlavorImageConflict: Memory encryption requested by hw:mem_encryption extra spec in m1.small-enc flavor but image None doesn't have 'hw_firmware_type' property set to 'uefi' or volume-backed instance was requested
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] Traceback (most recent call last):
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2615, in _build_and_run_instance
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] self.driver.spawn(context, instance, image_meta,
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4413, in spawn
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] xml = self._get_guest_xml(context, instance, network_info,
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7565, in _get_guest_xml
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] conf = self._get_guest_config(instance, network_info, image_meta,
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7045, in _get_guest_config
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] guest.cpu = self._get_guest_cpu_config(
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 5451, in _get_guest_cpu_config
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] cpu = self._get_guest_cpu_model_config(flavor, arch)
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 5398, in _get_guest_cpu_model_config
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] flags = libvirt_utils.get_flags_by_flavor_specs(flavor)
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/utils.py", line 658, in get_flags_by_flavor_specs
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] resource_request = scheduler_utils.ResourceRequest.from_request_spec(
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 187, in from_request_spec
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] res_req._translate_memory_encryption(request_spec.flavor, image)
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 297, in _translate_memory_encryption
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] if not hardware.get_mem_encryption_constraint(flavor, image):
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/hardware.py", line 1197, in get_mem_encryption_constraint
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] _check_mem_encryption_uses_uefi_image(requesters, image_meta)
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] File "/usr/lib/python3/dist-packages/nova/virt/hardware.py", line 1243, in _check_mem_encryption_uses_uefi_image
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] raise exception.FlavorImageConflict(emsg % data)
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542] nova.exception.FlavorImageConflict: Memory encryption requested by hw:mem_encryption extra spec in m1.small-enc flavor but image None doesn't have 'hw_firmware_type' property set to 'uefi' or volume-backed instance was requested
2023-10-25 06:33:20.674 38337 ERROR nova.compute.manager [instance: 000b22bc-6b28-4adb-a3af-44b1f090c542]
```

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

Ubuntu 22.04 and UCA bobcat.

# dpkg -l | grep nova
ii nova-api 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-common 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-conductor 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-novncproxy 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python3-nova 3:28.0.0-0ubuntu1~cloud0 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:18.4.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - 3.x

2. Which hypervisor did you use?
Libvirt + KVM

3. Which storage type did you use?
LVM

4. Which networking type did you use?
ml2 + ovs

[Takashi Kajinami (kajinamit)]

It seems the issue here is specific to the combination of flavor properties and image properties.
If I use hw_mem_encryption image property instead of hw:mem_encryption flavor property, the issue is not reproduced.

[Takashi Kajinami (kajinamit)]

I tried setting hw:irmware_type flavor property but I eventually noticed nova does not support this usage, which means that hw:mem_encryption flavor property is useless...

[Uggla (rene-ribaud)]

Hello Takashi, thanks for submitting this bug. I manage to reproduce it (devstack) as well.
Do you know if this was working with a previous version ?

[Takashi Kajinami (kajinamit)]

> Do you know if this was working with a previous version ?

Unfortunately no. I've never used this feature before and now I'm trying it with Bobcat release for my first time.

I'll attempt to prepare an env with an older version but it likely takes some time. It'd be nice if someone can look into the failing logic and invest if there are any recent changes possibly causing this.

[sean mooney (sean-k-mooney)]

not quite useless if you set hw:mem_encryption then you need to separately set hw_firmware_type in the image

historically configuration of the emulatated devices in a VM was only supported in the image
that also include which fireware interface bios or uefi is presented to the guest.

we could extend that to the flavor but the separation of image vs flavour has been
flavor is for qunatitive elements and image is for qulaitivate.

i can see a use case where an admin wants to create a uefi flavor but the other reason for not doing that in the past is it can only be used with a uefi capable image.
i.e. an image with a gpt portion table and an efi portion
that is why we have used the image to hold the annotation that declares it support uefi not the flavor.