OpenStack Compute (nova)

overlapping pinned CPUs after unshelve

Bug #2025480 reported by Bence Romsics
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Unassigned

Bug Description

It seems that after unshelve, occasionally the request for a dedicated CPU is ignored. More precisely the first pinned CPU does not seem to be marked as consumed, so the second may end up on the same CPU. This was first observed on victoria (6 times out of 46 tries), but then I was able to reproduce it on master too (6 times out of 20 tries). The logs attached are from the victoria environment, which was a single-host all-in-one devstack running only the vms used for this reprouduction.

stable/victoria
devstack 3eb6e2d7
nova 1aca09b966

master
devstack b10c0602
nova 2aea80c0af

config:
[[post-config|/etc/nova/nova.conf]]
[DEFAULT]
scheduler_default_filters = NUMATopologyFilter, ...
[compute]
cpu_dedicated_set = 0,1

Confirming this config in placement:
$ openstack --os-placement-api-version 1.17 resource provide inventory show 46b3d4de-bb45-4607-8860-040eb2dcd0d7 PCPU
+------------------+-------+
| Field | Value |
+------------------+-------+
| allocation_ratio | 1.0 |
| min_unit | 1 |
| max_unit | 2 |
| reserved | 0 |
| step_size | 1 |
| total | 2 |
+------------------+-------+

Reproduction steps:
openstack flavor create cirros256-pinned --public --vcpus 1 --ram 256 --disk 1 --property hw_rng:allowed=True --property hw:cpu_policy=dedicated

openstack server list -f value -c ID | xargs -r openstack server delete --wait

openstack server create --flavor cirros256-pinned --image cirros-0.5.1-x86_64-disk --nic net-id=private vm0 --wait
openstack server shelve vm0
sleep 10 # make sure shelve finished
openstack server create --flavor cirros256-pinned --image cirros-0.5.1-x86_64-disk --nic net-id=private vm1 --wait
openstack server shelve vm1
sleep 10

openstack server unshelve vm0 ; sleep 15 ; openstack server unshelve vm1 # the amount of sleep could easily be relevant
watch openstack server list # wait until both go ACTIVE

# both vms ended up on the same cpu
$ for vm in $( sudo virsh list --name ) ; do sudo virsh dumpxml $vm | xmlstarlet sel -t -v '//vcpupin/@cpuset' ; echo ; done
0
0

Data collected from the environment where the above reproduction triggered the bug:

$ openstack server list
+--------------------------------------+------+--------+--------------------------------------------------------+--------------------------+------------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+--------------------------------------------------------+--------------------------+------------------+
| 4734b8a5-a6dd-432a-86c9-ba0367bb86cc | vm1 | ACTIVE | private=10.0.0.27, fdfb:ab27:b2b2:0:f816:3eff:fe80:2fd | cirros-0.5.1-x86_64-disk | cirros256-pinned |
| e30de509-6988-4535-a6f5-520c52fba087 | vm0 | ACTIVE | private=10.0.0.6, fdfb:ab27:b2b2:0:f816:3eff:fe78:d368 | cirros-0.5.1-x86_64-disk | cirros256-pinned |
+--------------------------------------+------+--------+--------------------------------------------------------+--------------------------+------------------+

$ openstack server show vm0
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack1v |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack1v |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001f |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2023-06-29T10:45:25.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private=10.0.0.6, fdfb:ab27:b2b2:0:f816:3eff:fe78:d368 |
| config_drive | |
| created | 2023-06-29T10:44:42Z |
| flavor | cirros256-pinned (a61fac7f-9d3a-4788-8dd8-b84534ed5ba6) |
| hostId | a660a4498447edab1528a7c92f7eaccd7aad2c258c896f7820aef9f6 |
| id | e30de509-6988-4535-a6f5-520c52fba087 |
| image | cirros-0.5.1-x86_64-disk (aaefde72-6da8-45e3-bc58-d7d1b9333d0e) |
| key_name | None |
| name | vm0 |
| progress | 0 |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2023-06-29T10:45:26Z |
| user_id | 971105d66de542588ddccdef15594fab |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------------------------+

$ openstack server show vm1
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | devstack1v |
| OS-EXT-SRV-ATTR:hypervisor_hostname | devstack1v |
| OS-EXT-SRV-ATTR:instance_name | instance-00000020 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2023-06-29T10:45:45.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private=10.0.0.27, fdfb:ab27:b2b2:0:f816:3eff:fe80:2fd |
| config_drive | |
| created | 2023-06-29T10:45:00Z |
| flavor | cirros256-pinned (a61fac7f-9d3a-4788-8dd8-b84534ed5ba6) |
| hostId | a660a4498447edab1528a7c92f7eaccd7aad2c258c896f7820aef9f6 |
| id | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc |
| image | cirros-0.5.1-x86_64-disk (aaefde72-6da8-45e3-bc58-d7d1b9333d0e) |
| key_name | None |
| name | vm1 |
| progress | 0 |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2023-06-29T10:45:45Z |
| user_id | 971105d66de542588ddccdef15594fab |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------------------------+

$ openstack flavor show cirros256-pinned
+----------------------------+--------------------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| disk | 1 |
| id | a61fac7f-9d3a-4788-8dd8-b84534ed5ba6 |
| name | cirros256-pinned |
| os-flavor-access:is_public | True |
| properties | hw:cpu_policy='dedicated', hw_rng:allowed='True' |
| ram | 256 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------------------+

$ openstack server event list vm0
+------------------------------------------+--------------------------------------+----------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------+----------------------------+
| req-e09149a8-6ef9-49c3-a757-57bbc14d6004 | e30de509-6988-4535-a6f5-520c52fba087 | unshelve | 2023-06-29T10:45:19.000000 |
| req-4628f975-a57b-468e-a41c-0ef405df4bae | e30de509-6988-4535-a6f5-520c52fba087 | shelve | 2023-06-29T10:44:49.000000 |
| req-d7595149-3cc6-472c-9da8-7a40c020a620 | e30de509-6988-4535-a6f5-520c52fba087 | create | 2023-06-29T10:44:42.000000 |
+------------------------------------------+--------------------------------------+----------+----------------------------+
$ openstack server event show vm0 req-e09149a8-6ef9-49c3-a757-57bbc14d6004
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | unshelve |
| events | [{'event': 'compute_unshelve_instance', 'start_time': '2023-06-29T10:45:19.000000', 'finish_time': '2023-06-29T10:45:26.000000', 'result': 'Success', 'traceback': None}, {'event': 'schedule_instances', 'start_time': |
| | '2023-06-29T10:45:19.000000', 'finish_time': '2023-06-29T10:45:19.000000', 'result': 'Success', 'traceback': None}, {'event': 'get_image_info', 'start_time': '2023-06-29T10:45:19.000000', 'finish_time': |
| | '2023-06-29T10:45:19.000000', 'result': 'Success', 'traceback': None}] |
| instance_uuid | e30de509-6988-4535-a6f5-520c52fba087 |
| message | None |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| request_id | req-e09149a8-6ef9-49c3-a757-57bbc14d6004 |
| start_time | 2023-06-29T10:45:19.000000 |
| user_id | 971105d66de542588ddccdef15594fab |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack server event show vm0 req-4628f975-a57b-468e-a41c-0ef405df4bae
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | shelve |
| events | [{'event': 'compute_shelve_instance', 'start_time': '2023-06-29T10:44:49.000000', 'finish_time': '2023-06-29T10:44:55.000000', 'result': 'Success', 'traceback': None}] |
| instance_uuid | e30de509-6988-4535-a6f5-520c52fba087 |
| message | None |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| request_id | req-4628f975-a57b-468e-a41c-0ef405df4bae |
| start_time | 2023-06-29T10:44:49.000000 |
| user_id | 971105d66de542588ddccdef15594fab |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack server event list vm1
+------------------------------------------+--------------------------------------+----------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------+----------------------------+
| req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc | unshelve | 2023-06-29T10:45:35.000000 |
| req-1dc9b980-d765-415a-b946-2daf53f361b1 | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc | shelve | 2023-06-29T10:45:07.000000 |
| req-5071fd9f-00e2-44f8-9b6c-8715fe918e98 | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc | create | 2023-06-29T10:45:00.000000 |
+------------------------------------------+--------------------------------------+----------+----------------------------+
$ openstack server event show vm1 req-9ab0eabc-bf97-48bf-b534-dd22df3c6452
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | unshelve |
| events | [{'event': 'compute_unshelve_instance', 'start_time': '2023-06-29T10:45:36.000000', 'finish_time': '2023-06-29T10:45:45.000000', 'result': 'Success', 'traceback': None}, {'event': 'schedule_instances', 'start_time': |
| | '2023-06-29T10:45:35.000000', 'finish_time': '2023-06-29T10:45:36.000000', 'result': 'Success', 'traceback': None}, {'event': 'get_image_info', 'start_time': '2023-06-29T10:45:35.000000', 'finish_time': |
| | '2023-06-29T10:45:35.000000', 'result': 'Success', 'traceback': None}] |
| instance_uuid | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc |
| message | None |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| request_id | req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 |
| start_time | 2023-06-29T10:45:35.000000 |
| user_id | 971105d66de542588ddccdef15594fab |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack server event show vm1 req-1dc9b980-d765-415a-b946-2daf53f361b1
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | shelve |
| events | [{'event': 'compute_shelve_instance', 'start_time': '2023-06-29T10:45:08.000000', 'finish_time': '2023-06-29T10:45:13.000000', 'result': 'Success', 'traceback': None}] |
| instance_uuid | 4734b8a5-a6dd-432a-86c9-ba0367bb86cc |
| message | None |
| project_id | 0ad9573f6c57482bbd8202ce43706ca6 |
| request_id | req-1dc9b980-d765-415a-b946-2daf53f361b1 |
| start_time | 2023-06-29T10:45:07.000000 |
| user_id | 971105d66de542588ddccdef15594fab |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Logs that seemed relevant to me:
vm0 unshelve:
jún 29 10:45:19 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw={0=0},cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=VirtCPUTopology,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) on host_cell NUMACell(cpu_usage=0,cpuset=set([]),id=0,memory=7957,memory_usage=0,mempages=[NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([0,1]),pinned_cpus=set([]),siblings=[set([0]),set([1])]) {{(pid=379150) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1046}}
jún 29 10:45:19 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Pinning has been requested {{(pid=379150) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1133}}
jún 29 10:45:19 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Packing an instance onto a set of siblings: host_cell_free_siblings: [{0}, {1}] instance_cell: InstanceNUMACell(cpu_pinning_raw={0=0},cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=VirtCPUTopology,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) host_cell_id: 0 threads_per_core: 1 num_cpu_reserved: 0 {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:731}}
jún 29 10:45:19 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Built sibling_sets: defaultdict(<class 'list'>, {1: [{0}, {1}]}) {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:752}}
jún 29 10:45:19 devstack1v nova-compute[379150]: INFO nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1]], vCPUs mapping: [(0, 0)]
jún 29 10:45:19 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] Selected cores for pinning: [(0, 0)], in cell 0 {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:1013}}
jún 29 10:45:19 devstack1v nova-compute[379150]: INFO nova.compute.claims [None req-e09149a8-6ef9-49c3-a757-57bbc14d6004 admin admin] [instance: e30de509-6988-4535-a6f5-520c52fba087] Claim successful on node devstack1v

vm1 unshelve:
jún 29 10:45:37 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw={0=0},cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=VirtCPUTopology,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) on host_cell NUMACell(cpu_usage=0,cpuset=set([]),id=0,memory=7957,memory_usage=0,mempages=[NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([0,1]),pinned_cpus=set([]),siblings=[set([0]),set([1])]) {{(pid=379150) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1046}}
jún 29 10:45:37 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Pinning has been requested {{(pid=379150) _numa_fit_instance_cell /opt/stack/nova/nova/virt/hardware.py:1133}}
jún 29 10:45:37 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Packing an instance onto a set of siblings: host_cell_free_siblings: [{0}, {1}] instance_cell: InstanceNUMACell(cpu_pinning_raw={0=0},cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=VirtCPUTopology,cpuset=set([]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([0])) host_cell_id: 0 threads_per_core: 1 num_cpu_reserved: 0 {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:731}}
jún 29 10:45:37 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Built sibling_sets: defaultdict(<class 'list'>, {1: [{0}, {1}]}) {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:752}}
jún 29 10:45:37 devstack1v nova-compute[379150]: INFO nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1]], vCPUs mapping: [(0, 0)]
jún 29 10:45:37 devstack1v nova-compute[379150]: DEBUG nova.virt.hardware [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] Selected cores for pinning: [(0, 0)], in cell 0 {{(pid=379150) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:1013}}
jún 29 10:45:37 devstack1v nova-compute[379150]: INFO nova.compute.claims [None req-9ab0eabc-bf97-48bf-b534-dd22df3c6452 admin admin] [instance: 4734b8a5-a6dd-432a-86c9-ba0367bb86cc] Claim successful on node devstack1v

The first error in the logs came not during unshelve but well after:
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Error updating resources for node devstack1v.: nova.exception.CPUPinningInvalid: CPU set to pin [0] must
 be a subset of free CPU set [1]
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager Traceback (most recent call last):
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/manager.py", line 9973, in _update_available_resource_for_node
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager self.rt.update_available_resource(context, nodename,
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 896, in update_available_resource
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager return f(*args, **kwargs)
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 968, in _update_available_resource
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager instance_by_uuid = self._update_usage_from_instances(
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1534, in _update_usage_from_instances
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager self._update_usage_from_instance(context, instance, nodename)
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1497, in _update_usage_from_instance
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager self._update_usage(self._get_usage_dict(instance, instance),
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/compute/resource_tracker.py", line 1301, in _update_usage
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager cn.numa_topology = hardware.numa_usage_from_instance_numa(
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/virt/hardware.py", line 2450, in numa_usage_from_instance_numa
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager new_cell.pin_cpus(pinned_cpus)
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager File "/opt/stack/nova/nova/objects/numa.py", line 91, in pin_cpus
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager raise exception.CPUPinningInvalid(requested=list(cpus),
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager nova.exception.CPUPinningInvalid: CPU set to pin [0] must be a subset of free CPU set [1]
jún 29 10:47:27 devstack1v nova-compute[379150]: ERROR nova.compute.manager
jún 29 10:47:28 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager._instance_usage_audit {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}

Between the two unshelves the update_available_resource task ran once:
$ egrep 'Running periodic task ComputeManager.update_available_resource' ~/n-cpu.log
jún 29 10:35:16 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:36:17 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:37:17 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:38:19 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:39:20 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:40:22 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:41:22 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:42:23 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:43:24 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:44:24 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:45:24 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:46:26 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:47:27 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:48:28 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}
jún 29 10:49:28 devstack1v nova-compute[379150]: DEBUG oslo_service.periodic_task [None req-a52cbcd3-ad79-457e-a0e1-3ec225226411 None None] Running periodic task ComputeManager.update_available_resource {{(pid=379150) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}}

I also attached a .tar.gz with the following contents:

for service in n-api n-cond-cell1 n-cpu n-sch n-super-cond
do
    sudo journalctl -u devstack@$service -S '2023-06-29 10:35:00' -U '2023-06-29 10:50:00' > ~/$service.log
done

$ sudo virsh list
 Id Name State
-----------------------------------
 63 instance-0000001f running
 64 instance-00000020 running

sudo virsh dumpxml instance-0000001f > ~/instance-0000001f.xml
sudo virsh dumpxml instance-00000020 > ~/instance-00000020.xml

Let me know if I can collect more information.

Revision history for this message
Bence Romsics (bence-romsics) wrote :
Revision history for this message
Bence Romsics (bence-romsics) wrote :
Download full text (3.4 KiB)

By the way here is the state of placement allocations when the bug is present (on master):

$ openstack resource provider list
+--------------------------------------+-----------+------------+--------------------------------------+----------------------+
| uuid | name | generation | root_provider_uuid | parent_provider_uuid |
 +--------------------------------------+-----------+------------+--------------------------------------+----------------------+
| 513a47b7-38af-4fa0-9199-3f6d16d600a6 | devstack0 | 108 | 513a47b7-38af-4fa0-9199-3f6d16d600a6 | None |
 +--------------------------------------+-----------+------------+--------------------------------------+----------------------+
$ openstack resource provider show --allocations 513a47b7-38af-4fa0-9199-3f6d16d600a6
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
 +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| uuid | 513a47b7-38af-4fa0-9199-3f6d16d600a6 |
| name | devstack0 |
| generation | 108 |
| root_provider_uuid | 513a47b7-38af-4fa0-9199-3f6d16d600a6 |
| parent_provider_uuid | None |
| allocations | {'29ea91aa-c1eb-4ea6-bf1a-6459b68be55a': {'resources': {'DISK_GB': 1, 'MEMORY_MB': 256, 'PCPU': 1}, 'consumer_generation': 1}, '6bdb6725-8c3e-422d-8c95-1b7459b2f4bd': {'resources': {'DISK_GB': 1, 'MEMORY_MB': |
| | 256, 'PCPU': 1}, 'consumer_generation': 1}} ...

Read more...

Revision history for this message
Bence Romsics (bence-romsics) wrote :

I believe this is not a recurrence of https://bugs.launchpad.net/nova/+bug/1988311, because I don't see any overlap in this: egrep -o 'Lock "compute_resources" (acquired|released)' n-cpu.log

Revision history for this message
Bence Romsics (bence-romsics) wrote :

I believe I found where we store the pcpu usage info the nova cell db, so I reproduced the bug while saving the relevant db column's value. This is what I did:

function save-numa-topology () {
    mysql --skip-column-names nova_cell1 <<<"select numa_topology from compute_nodes" | json_pp > "numa-topology.$( date --rfc-3339=s | tr ' ' T ).json"
}

source ~/src/os/openstack/devstack/openrc admin admin >/dev/null

openstack server list -f value -c ID | xargs -r openstack server delete --wait

openstack server create --flavor "$flavor" --image "$image" --nic net-id="$net" vm0 --wait >/dev/null
openstack server shelve vm0
sleep 10
openstack server create --flavor "$flavor" --image "$image" --nic net-id="$net" vm1 --wait >/dev/null
openstack server shelve vm1
sleep 10

save-numa-topology
openstack server unshelve vm0
sleep 15
save-numa-topology
openstack server unshelve vm1

while [ $( openstack server list -f value -c Name -c Status | egrep -w 'vm[01]' | egrep -cw ACTIVE ) != 2 ]
do
    sleep 1
done
save-numa-topology

I attached another .tar.gz with 3 files containing the value of numa_topology from the above run. This time I reproduced the bug on the master branch.

Revision history for this message
Bence Romsics (bence-romsics) wrote :

By the way another possible side effect of this bug is that vm deletion my fail for vms whose pinned cpus are incorrectly tracked with an error like this: {'code': 400, 'created': '2023-07-06T12:00:29Z', 'message': 'CPU set to unpin [0] must be a subset of known CPU set [2, 3]'}

Revision history for this message
Bence Romsics (bence-romsics) wrote (last edit ):

I had an idea to measure the frequency of this bug with different update_resources_interval settings (fails/tries):

update_resources_interval = -1 : 0/40
update_resources_interval = 120 : 2/40
update_resources_interval = 60 : 4/20
update_resources_interval = 20 : 11/20

This makes me think that something triggered by api operations is racing with something in the update_available_resource periodic job.

Revision history for this message
Bence Romsics (bence-romsics) wrote (last edit ):

It seems that when we fail we have a log message like this:

2023-07-12T11:36:01.045934+0000 devstack0 nova-compute[358332]: DEBUG nova.compute.resource_tracker [None req-6880982f-bd20-4983-b212-4ce32465fdd7 None None] [instance: ad7997b9-1c2c-4776-a30b-0f2b62fc2222] Instance with task_state "spawning" is not being actively managed by this compute host but has allocations referencing this compute node (3698a4ab-4810-4beb-915a-ed3ed883a2e1): {'resources': {'DISK_GB': 1, 'MEMORY_MB': 256, 'PCPU': 1}}. Skipping heal of allocations during the task state transition. {{(pid=358332) _remove_deleted_instances_allocations /opt/stack/nova/nova/compute/resource_tracker.py:1717}}

Which originates from here:

https://opendev.org/openstack/nova/src/commit/6f56c5c9fd60ee1d53376a9100a9580cb2b38dc3/nova/compute/resource_tracker.py#L1707-L1730

That comment sounds suspiciously relevant to this bug. However not being familiar with nova internals, I'm far from properly understanding at the moment in what conditions we get here and why this happens.

edit: This log message is not present every time the bug is present, so the above may be a false suspicion.

Revision history for this message
Uggla (rene-ribaud) wrote :

Hello, thanks for reporting this bug with so many details. This is well appreciated.
I set this bug to triage as it looks valid to me, although I have not figured out why it is assigning CPUs wrongly.
I will share this bug with team members with more experience, maybe they will have an idea.

Changed in nova:
status: New → Triaged
Revision history for this message
Pawel Koniszewski (pawel-koniszewski) wrote (last edit ):

It seems to me that it is due to this if condition racing with unshelve operation:

https://github.com/openstack/nova/blob/f0565e84ee9578d6dafd22d57fb0c95cb3984c1e/nova/compute/resource_tracker.py#L1673

When unshelving operation is on-going, the VM may be in SHELVE_OFFLOADED state, however, its task state may be already set to SPAWNING. The instance claim happens when VM is in SHELVE_OFFLOADED/SPAWNING state:

https://github.com/openstack/nova/blob/f0565e84ee9578d6dafd22d57fb0c95cb3984c1e/nova/compute/manager.py#L7075

so when update_available_resources is triggered the if condition does not take this instance usage into consideration effectively removing resources claimed by the instance.

Revision history for this message
Bence Romsics (bence-romsics) wrote :

I hope I understood the root cause, so I proposed a fix, please have a look here:
https://review.opendev.org/c/openstack/nova/+/890406

Revision history for this message
Bence Romsics (bence-romsics) wrote :

Btw for the ease of reproduction:

I usually changed some periodic job intervals to make the bug more frequent:

/etc/nova/nova-cpu.conf
[DEFAULT]
update_resources_interval = 60

And here's my final bug reproducer script (beware, it deletes all servers):

#! /bin/bash

flavor="cirros256-pinned"
image="cirros-0.5.2-x86_64-disk"
net="private"

source ~/src/os/openstack/devstack/openrc admin admin >/dev/null

# Delete twice because vms in the error state may need that.
openstack server list -f value -c ID | xargs -r openstack server delete --wait
openstack server list -f value -c ID | xargs -r openstack server delete --wait

since="$( date '+%Y-%m-%d %H:%M:%S' )"

openstack server create --flavor "$flavor" --image "$image" --nic net-id="$net" vm0 --wait >/dev/null
openstack server shelve vm0
sleep 10
openstack server create --flavor "$flavor" --image "$image" --nic net-id="$net" vm1 --wait >/dev/null
openstack server shelve vm1
sleep 10

openstack server unshelve vm0
sleep 15
openstack server unshelve vm1

while [ $( openstack server list -f value -c Name -c Status | egrep -w 'vm[01]' | egrep -cw ACTIVE ) != 2 ]
do
    sleep 1
done

until="$( date '+%Y-%m-%d %H:%M:%S' )"

# This way we only catch resource tracking errors during unshelve of vm0.
# The same resource tracking error may happen during unshelve of vm1,
# but we would need to start one more vm to catch that...
cpuset0="$( sudo virsh dumpxml $( openstack server show vm0 -f value -c OS-EXT-SRV-ATTR:instance_name ) | xmlstarlet sel -t -v '//vcpupin/@cpuset' )"
cpuset1="$( sudo virsh dumpxml $( openstack server show vm1 -f value -c OS-EXT-SRV-ATTR:instance_name ) | xmlstarlet sel -t -v '//vcpupin/@cpuset' )"
if [ "$cpuset0" = "$cpuset1" ]
then
    exit_msg=fail
    exit_code=1
else
    exit_msg=pass
    exit_code=0
fi

#echo -S "'$since'" -U "'$until'"

#echo '>>>'
#sudo journalctl -u devstack@n-cpu -S "$since" -U "$until" | egrep 'ERROR|AAA'
#echo '<<<'

echo >&2 "$exit_msg"
exit $exit_code

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/891290

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/891290
Committed: https://opendev.org/openstack/nova/commit/62300d4885549368f874b3e07b756017ff96c659
Submitter: "Zuul (22348)"
Branch: master

commit 62300d4885549368f874b3e07b756017ff96c659
Author: Bence Romsics <email address hidden>
Date: Mon Aug 14 13:03:13 2023 +0200

    Reproduce bug #2025480 in a functional test

    Written by gibi, I just cleaned it up.

    Change-Id: I8386a846b3685b8d03c59334ccfb2efbd4afe427
    Co-Authored-By: Balazs Gibizer <email address hidden>
    Related-Bug: #2025480

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/890406
Committed: https://opendev.org/openstack/nova/commit/f1dc4ec39bcfda1bd4b97e233a9da498b6378c4f
Submitter: "Zuul (22348)"
Branch: master

commit f1dc4ec39bcfda1bd4b97e233a9da498b6378c4f
Author: Bence Romsics <email address hidden>
Date: Wed Aug 2 16:22:55 2023 +0200

    Do not untrack resources of a server being unshelved

    This patch concerns the time when a VM is being unshelved and the
    compute manager set the task_state to spawning, claimed resources of
    the VM and then called driver.spawn(). So the instance is in vm_state
    SHELVED_OFFLOADED, task_state spawning.

    If at this point a new update_available_resource periodic job is
    started that collects all the instances assigned to the node to
    calculate resource usage. However the calculation assumed that a
    VM in SHELVED_OFFLOADED state does not need resource allocation on
    the node (probably being removed from the node as it is offloaded)
    and deleted the resource claim.

    Given all this we ended up with the VM spawned successfully but having
    lost the resource claim on the node.

    This patch changes what we do in vm_state SHELVED_OFFLOADED, task_state
    spawning. We no longer delete the resource claim in this state and
    keep tracking the resource in stats.

    Change-Id: I8c9944810c09d501a6d3f60f095d9817b756872d
    Closes-Bug: #2025480

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/2023.2)

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/904370

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/904371

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/2023.1)

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/904372

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/904373

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/904374

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/904375

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904370
Committed: https://opendev.org/openstack/nova/commit/477ff2667d7ecd218fa5163d86d2719979dcdcd3
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 477ff2667d7ecd218fa5163d86d2719979dcdcd3
Author: Bence Romsics <email address hidden>
Date: Mon Aug 14 13:03:13 2023 +0200

    Reproduce bug #2025480 in a functional test

    Written by gibi, I just cleaned it up.

    Change-Id: I8386a846b3685b8d03c59334ccfb2efbd4afe427
    Co-Authored-By: Balazs Gibizer <email address hidden>
    Related-Bug: #2025480
    (cherry picked from commit 62300d4885549368f874b3e07b756017ff96c659)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904371
Committed: https://opendev.org/openstack/nova/commit/4239d1fec2814c074482b740a2fd38a5d5ce6942
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 4239d1fec2814c074482b740a2fd38a5d5ce6942
Author: Bence Romsics <email address hidden>
Date: Wed Aug 2 16:22:55 2023 +0200

    Do not untrack resources of a server being unshelved

    This patch concerns the time when a VM is being unshelved and the
    compute manager set the task_state to spawning, claimed resources of
    the VM and then called driver.spawn(). So the instance is in vm_state
    SHELVED_OFFLOADED, task_state spawning.

    If at this point a new update_available_resource periodic job is
    started that collects all the instances assigned to the node to
    calculate resource usage. However the calculation assumed that a
    VM in SHELVED_OFFLOADED state does not need resource allocation on
    the node (probably being removed from the node as it is offloaded)
    and deleted the resource claim.

    Given all this we ended up with the VM spawned successfully but having
    lost the resource claim on the node.

    This patch changes what we do in vm_state SHELVED_OFFLOADED, task_state
    spawning. We no longer delete the resource claim in this state and
    keep tracking the resource in stats.

    Change-Id: I8c9944810c09d501a6d3f60f095d9817b756872d
    Closes-Bug: #2025480
    (cherry picked from commit f1dc4ec39bcfda1bd4b97e233a9da498b6378c4f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904372
Committed: https://opendev.org/openstack/nova/commit/23c190a35839b396418d3e98af1e67587f9e9296
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 23c190a35839b396418d3e98af1e67587f9e9296
Author: Bence Romsics <email address hidden>
Date: Mon Aug 14 13:03:13 2023 +0200

    Reproduce bug #2025480 in a functional test

    Written by gibi, I just cleaned it up.

    Change-Id: I8386a846b3685b8d03c59334ccfb2efbd4afe427
    Co-Authored-By: Balazs Gibizer <email address hidden>
    Related-Bug: #2025480
    (cherry picked from commit 62300d4885549368f874b3e07b756017ff96c659)
    (cherry picked from commit 477ff2667d7ecd218fa5163d86d2719979dcdcd3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904373
Committed: https://opendev.org/openstack/nova/commit/683ecc060e3bca818b9fb514d297e323bc8cb220
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 683ecc060e3bca818b9fb514d297e323bc8cb220
Author: Bence Romsics <email address hidden>
Date: Wed Aug 2 16:22:55 2023 +0200

    Do not untrack resources of a server being unshelved

    This patch concerns the time when a VM is being unshelved and the
    compute manager set the task_state to spawning, claimed resources of
    the VM and then called driver.spawn(). So the instance is in vm_state
    SHELVED_OFFLOADED, task_state spawning.

    If at this point a new update_available_resource periodic job is
    started that collects all the instances assigned to the node to
    calculate resource usage. However the calculation assumed that a
    VM in SHELVED_OFFLOADED state does not need resource allocation on
    the node (probably being removed from the node as it is offloaded)
    and deleted the resource claim.

    Given all this we ended up with the VM spawned successfully but having
    lost the resource claim on the node.

    This patch changes what we do in vm_state SHELVED_OFFLOADED, task_state
    spawning. We no longer delete the resource claim in this state and
    keep tracking the resource in stats.

    Change-Id: I8c9944810c09d501a6d3f60f095d9817b756872d
    Closes-Bug: #2025480
    (cherry picked from commit f1dc4ec39bcfda1bd4b97e233a9da498b6378c4f)
    (cherry picked from commit 4239d1fec2814c074482b740a2fd38a5d5ce6942)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904374
Committed: https://opendev.org/openstack/nova/commit/7422642dd6e222959470b73dc6ba2450a792b9c7
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 7422642dd6e222959470b73dc6ba2450a792b9c7
Author: Bence Romsics <email address hidden>
Date: Mon Aug 14 13:03:13 2023 +0200

    Reproduce bug #2025480 in a functional test

    Written by gibi, I just cleaned it up.

    Change-Id: I8386a846b3685b8d03c59334ccfb2efbd4afe427
    Co-Authored-By: Balazs Gibizer <email address hidden>
    Related-Bug: #2025480
    (cherry picked from commit 62300d4885549368f874b3e07b756017ff96c659)
    (cherry picked from commit 477ff2667d7ecd218fa5163d86d2719979dcdcd3)
    (cherry picked from commit 23c190a35839b396418d3e98af1e67587f9e9296)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/nova/+/904375
Committed: https://opendev.org/openstack/nova/commit/e41962f5fa59c47e63468945e82c2e7164c24c38
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e41962f5fa59c47e63468945e82c2e7164c24c38
Author: Bence Romsics <email address hidden>
Date: Wed Aug 2 16:22:55 2023 +0200

    Do not untrack resources of a server being unshelved

    This patch concerns the time when a VM is being unshelved and the
    compute manager set the task_state to spawning, claimed resources of
    the VM and then called driver.spawn(). So the instance is in vm_state
    SHELVED_OFFLOADED, task_state spawning.

    If at this point a new update_available_resource periodic job is
    started that collects all the instances assigned to the node to
    calculate resource usage. However the calculation assumed that a
    VM in SHELVED_OFFLOADED state does not need resource allocation on
    the node (probably being removed from the node as it is offloaded)
    and deleted the resource claim.

    Given all this we ended up with the VM spawned successfully but having
    lost the resource claim on the node.

    This patch changes what we do in vm_state SHELVED_OFFLOADED, task_state
    spawning. We no longer delete the resource claim in this state and
    keep tracking the resource in stats.

    Conflicts:
        nova/compute/vm_states.py
            conflict with 8c2e76598995f0d417653c60a63ea342baf4e880

    Change-Id: I8c9944810c09d501a6d3f60f095d9817b756872d
    Closes-Bug: #2025480
    (cherry picked from commit f1dc4ec39bcfda1bd4b97e233a9da498b6378c4f)
    (cherry picked from commit 4239d1fec2814c074482b740a2fd38a5d5ce6942)
    (cherry picked from commit 683ecc060e3bca818b9fb514d297e323bc8cb220)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 26.2.2

This issue was fixed in the openstack/nova 26.2.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 29.0.0.0rc1

This issue was fixed in the openstack/nova 29.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 27.3.0

This issue was fixed in the openstack/nova 27.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 28.1.0

This issue was fixed in the openstack/nova 28.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.