OpenStack Compute (nova)

[SR-IOV][CPU Pinning] nova compute can try to boot VM with CPUs from one NUMA node and PCI device from another NUMA node.

Bug #1582278 reported by Sergey Nikitin
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Sergey Nikitin
Ubuntu Cloud Archive
Invalid
Undecided
Unassigned
Mitaka
Fix Released
Undecided
Unassigned
nova (Ubuntu)
Fix Released
High
Unassigned
Xenial
Fix Released
High
Unassigned
Yakkety
Fix Released
High
Unassigned

Bug Description

Environment:
Two NUMA nodes on compute host (node-0 and node-1).
One SR-IOV PCI device associated with NUMA node-1.

Steps to reproduce:

Steps to reproduce:
 1) Deploy env with SR-IOV and CPU pinning enable
 2) Create new flavor with cpu pinning:
nova flavor-show m1.small.performance
+----------------------------+-------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------------+-------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 20 |
| extra_specs | {"hw:cpu_policy": "dedicated", "hw:numa_nodes": "1"} |
| id | 7b0e5ee0-0bf7-4a46-9653-9279a947c650 |
| name | m1.small.performance |
| os-flavor-access:is_public | True |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------------------------------------------------
 3) download ubuntu image
 4) create sr-iov port and boot vm on this port with m1.small.performance flavor:
NODE_1='node-4.test.domain.local'
NODE_2='node-5.test.domain.local'
NET_ID_1=$(neutron net-list | grep net_EW_2 | awk '{print$2}')
neutron port-create $NET_ID_1 --binding:vnic-type direct --device_owner nova-compute --name sriov_23
port_id=$(neutron port-list | grep 'sriov_23' | awk '{print$2}')
nova boot vm23 --flavor m1.small.performance --image ubuntu_image --availability-zone nova:$NODE_1 --nic port-id=$port_id --key-name vm_key

Expected results:
 VM is an ACTIVE state
Actual result:
 In most cases the state is ERROR with following logs:

2016-05-13 08:25:56.598 29097 ERROR nova.pci.stats [req-26138c0b-fa55-4ff8-8f3a-aad980e3c815 d864c4308b104454b7b46fb652f4f377 9322dead0b5d440986b12596d9cbff5b - - -] Failed to allocate PCI devices for instance. Unassigning devices back to pools. This should not happen, since the scheduler should have accurate information, and allocation during claims is controlled via a hold on the compute node semaphore
2016-05-13 08:25:57.502 29097 INFO nova.virt.libvirt.driver [req-26138c0b-fa55-4ff8-8f3a-aad980e3c815 d864c4308b104454b7b46fb652f4f377 9322dead0b5d440986b12596d9cbff5b - - -] [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] Creating image
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager [req-26138c0b-fa55-4ff8-8f3a-aad980e3c815 d864c4308b104454b7b46fb652f4f377 9322dead0b5d440986b12596d9cbff5b - - -] Instance failed network setup after 1 attempt(s)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager Traceback (most recent call last):
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 666, in allocate_for_instance
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager self.force_reraise()
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 645, in allocate_for_instance
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 738, in _populate_neutron_extension_values
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager port_req_body)
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 709, in _populate_neutron_binding_profile
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager instance, pci_request_id).pop()
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager IndexError: pop from empty list
2016-05-13 08:25:57.664 29097 ERROR nova.compute.manager
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [req-26138c0b-fa55-4ff8-8f3a-aad980e3c815 d864c4308b104454b7b46fb652f4f377 9322dead0b5d440986b12596d9cbff5b - - -] [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] Instance failed to spawn
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] Traceback (most recent call last):
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2218, in _build_resources
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] yield resources
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] block_device_info=block_device_info)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] admin_pass=admin_password)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3287, in _create_image
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] instance, network_info, admin_pass, files, suffix)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3066, in _inject_data
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] network_info, libvirt_virt_type=CONF.libvirt.virt_type)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/virt/netutils.py", line 78, in get_injected_network_template
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] if not (network_info and template):
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 517, in __len__
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] return self._sync_wrapper(fn, *args, **kwargs)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 504, in _sync_wrapper
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] self.wait()
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 536, in wait
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] self[:] = self._gt.wait()
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] return self._exit_event.wait()
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] current.throw(*self._exc)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] result = function(*args, **kwargs)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1145, in context_wrapper
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] return func(*args, **kwargs)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1587, in _allocate_network_async
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] six.reraise(*exc_info)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] bind_host_id=bind_host_id)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 666, in allocate_for_instance
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] self._delete_ports(neutron, instance, created_port_ids)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] self.force_reraise()
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] six.reraise(self.type_, self.value, self.tb)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 645, in allocate_for_instance
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] bind_host_id=bind_host_id)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 738, in _populate_neutron_extension_values
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] port_req_body)
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 709, in _populate_neutron_binding_profile
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] instance, pci_request_id).pop()
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] IndexError: pop from empty list
2016-05-13 08:25:57.937 29097 ERROR nova.compute.manager [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566]
2016-05-13 08:25:57.939 29097 INFO nova.compute.manager [req-26138c0b-fa55-4ff8-8f3a-aad980e3c815 d864c4308b104454b7b46fb652f4f377 9322dead0b5d440986b12596d9cbff5b - - -] [instance: 4e691469-893d-4b24-a0a8-00bbee0fa566] Terminating instance

The problem is in nova/compute/resource_tracker.py. In method instance_claim():

claim = claims.Claim(context, instance_ref, self, self.compute_node,
                             overhead=overhead, limits=limits)
if self.pci_tracker:
  self.pci_tracker.claim_instance(context, instance_ref)

  instance_ref.numa_topology = claim.claimed_numa_topology
  self._set_instance_host_and_node(instance_ref)

1) here nova create a claim with correct NUMA node with CPU pinning and PCI devices (in our case it is node-1)
2) nova call pci_tracker.claim_instance() with instance_ref BUT instance_ref does not contain information about needed NUMA node. That is why in claim_instance we choose node-0. In this case we can't associate requested PCI devices with the instance because these devices are associated with node-1.
3) nova put to the instance_ref correct numa node-1 from step 1.
4) we got an instance without PCI devices.

See original description
Sergey Nikitin (snikitin)
Changed in nova:
status: New → Confirmed
description: updated
description: updated
Revision history for this message
Sergey Nikitin (snikitin) wrote :

Fixed by https://review.openstack.org/#/c/301308

Changed in nova:
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/317064

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/317065

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

As we use the "direct-release" model in Nova we don't use the
"Fix Comitted" status for merged bug fixes anymore. I'm setting
this manually to "Fix Released" to be consistent.

[1] "[openstack-dev] [release][all] bugs will now close automatically
    when patches merge"; Doug Hellmann; 2015-12-07;
    http://lists.openstack.org/pipermail/openstack-dev/2015-December/081612.html

Changed in nova:
status: Fix Committed → Fix Released
James Page (james-page)
tags: added: mitaka-backport-potential
Changed in nova (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → High
Changed in nova (Ubuntu Yakkety):
importance: Undecided → High
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/317064
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=257cfb7e6f2f3414640c632909f78db6b71f40b3
Submitter: Jenkins
Branch: stable/mitaka

commit 257cfb7e6f2f3414640c632909f78db6b71f40b3
Author: Jay Pipes <email address hidden>
Date: Fri Apr 1 16:03:47 2016 -0700

    pci: pass in instance PCI requests to claim

    Removes the calls to InstancePCIRequests.get_XXX() from within the
    claims.Claim and claims.MoveClaim constructors and instead has the
    resource tracker construct the PCI requests and pass them into the
    constructor.

    This allows us to remove the needlessly duplicative _test_pci() method
    in claims.MoveClaim and will allow the next patch in the series to
    remove the call in nova.pci.manager.PciDevTracker.claim_instance() that
    re-fetches PCI requests for the supplied instance.

    Related-Bug: #1368201
    Related-Bug: #1582278

    Change-Id: Ib2cc7c985839fbf88b5e6e437c4b395ab484b1b6
    (cherry picked from commit 74fbff88639891269f6a0752e70b78340cf87e9a)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/317065
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2150d5d8e17323bc9fce11903da3afffda211d26
Submitter: Jenkins
Branch: stable/mitaka

commit 2150d5d8e17323bc9fce11903da3afffda211d26
Author: Jay Pipes <email address hidden>
Date: Mon Apr 4 12:32:56 2016 -0700

    pci: eliminate DB lookup PCI requests during claim

    The nova.pci.manager.PciDevTracker.claim_instance() accepted an Instance
    object and called nova.objects.InstancePCIRequests.get_by_instance() to
    retrieve the PCI requests for the instance. This caused a DB lookup of
    the PCI requests for that instance, even though in all situations other
    than for migration/resize, the instance's PCI requests were already
    retrieved by the resource tracker.

    This change removes that additional DB lookup during claim_instance() by
    changing the instance parameter to instead be an InstancePCIRequests
    object and an InstanceNUMATopology object.

    Also in this patch is a change to nova.objects.PciDevice.claim() that
    changes the single parameter to an instance UUID instead of an Instance
    object, since nothing other than the instance's UUID was used in the
    method.

    Closes-Bug: #1582278
    Related-Bug: #1368201

    Change-Id: I9ab10c3035628f083233114b47b43a9b9ecdd166
    (cherry picked from commit 1f259e2a9423a4777f79ca561d5e6a74747a5019)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 13.1.2

This issue was fixed in the openstack/nova 13.1.2 release.

JuanJo Ciarlante (jjo)
tags: added: canonical-bootstack
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Marking as fix released for xenial/mitaka since 13.1.2 is now in xenial-updates and mitaka-updates.

Changed in cloud-archive:
status: New → Invalid
Changed in nova (Ubuntu Xenial):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.