Port update exception on nova unshelve for instance with PCI devices

[Steven Webster]

Description
===========
If an instance with PCI devices (SRIOV, or passthrough) is shelved, a port update exception will be seen and the instance will go into Error state when it is unshelved.

The nova API exception message is similar to:

"Unable to correlate PCI slot 0000:0d:00.1"

Steps to reproduce
==================
1. Launch an instance with SRIOV or PCI passthrough port bindings.

2. nova shelve <instance_uuid>

-- wait for nova instance status SHELVED_OFFLOADED --

3. nova unshelve <instance_uuid>

Expected result
===============
If there are resources available, the instance should be able to claim PCI devices and successfully (re)launch.

Actual result
=============
- Instance in error state
- Exception in nova api logs.

Environment
===========
1. Exact version of OpenStack you are running: Ocata, devstack

2. Which hypervisor did you use? Libvirt + KVM

2. Which storage type did you use? LVM

3. Which networking type did you use? Neutron, OVS

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/453938

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453938
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c2ff276c841934ff147aab836a4bd099297fb46b
Submitter: Jenkins
Branch: master

commit c2ff276c841934ff147aab836a4bd099297fb46b
Author: Steven Webster <email address hidden>
Date: Mon Mar 27 12:18:23 2017 -0400

    Fix port update exception when unshelving an instance with PCI devices

    It is possible that _update_port_binding_for_instance() is called
    without a migration object, such as when a user unshelves an instance.

    If the instance has a port(s) with a PCI device binding, the current
    logic extracts a pci mapping from old to new devices from the migration
    object and migration context. If a 'new' device is not found in the
    PCI mapping, an exception is thrown.

    In the case of an unshelve, there is no migration object (or migration
    context), and as such we have an empty pci mapping.

    This fix will only check for a new device if we have a migration object.

    Closes-Bug: 1677621
    Change-Id: I578153ca862753ef5b8041ee3853d3c7b2e2be30

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/459840

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/460173

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/460233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459840
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f281c18ba9aea1b2e8a36d5ae91a7acc5324ac5e
Submitter: Jenkins
Branch: stable/ocata

commit f281c18ba9aea1b2e8a36d5ae91a7acc5324ac5e
Author: Steven Webster <email address hidden>
Date: Mon Mar 27 12:18:23 2017 -0400

    Fix port update exception when unshelving an instance with PCI devices

    It is possible that _update_port_binding_for_instance() is called
    without a migration object, such as when a user unshelves an instance.

    If the instance has a port(s) with a PCI device binding, the current
    logic extracts a pci mapping from old to new devices from the migration
    object and migration context. If a 'new' device is not found in the
    PCI mapping, an exception is thrown.

    In the case of an unshelve, there is no migration object (or migration
    context), and as such we have an empty pci mapping.

    This fix will only check for a new device if we have a migration object.

    Closes-Bug: 1677621
    Change-Id: I578153ca862753ef5b8041ee3853d3c7b2e2be30
    (cherry picked from commit c2ff276c841934ff147aab836a4bd099297fb46b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460173
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=70c1eb689ad174b61ad915ae5384778bd536c16c
Submitter: Jenkins
Branch: stable/newton

commit 70c1eb689ad174b61ad915ae5384778bd536c16c
Author: Steven Webster <email address hidden>
Date: Mon Mar 27 12:18:23 2017 -0400

    Fix port update exception when unshelving an instance with PCI devices

    It is possible that _update_port_binding_for_instance() is called
    without a migration object, such as when a user unshelves an instance.

    If the instance has a port(s) with a PCI device binding, the current
    logic extracts a pci mapping from old to new devices from the migration
    object and migration context. If a 'new' device is not found in the
    PCI mapping, an exception is thrown.

    In the case of an unshelve, there is no migration object (or migration
    context), and as such we have an empty pci mapping.

    This fix will only check for a new device if we have a migration object.

    Conflicts:
          nova/tests/unit/network/test_neutronv2.py

    NOTE(mriedem): The conflict is due to not having change
    I818d2232f3398489be6303414585840c151e4db7 in Newton.

    Closes-Bug: 1677621
    Change-Id: I578153ca862753ef5b8041ee3853d3c7b2e2be30
    (cherry picked from commit c2ff276c841934ff147aab836a4bd099297fb46b)
    (cherry picked from commit f281c18ba9aea1b2e8a36d5ae91a7acc5324ac5e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460233
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=44a55d65e2aecdaa8fbcd32b2cde1b81a6fe74d8
Submitter: Jenkins
Branch: master

commit 44a55d65e2aecdaa8fbcd32b2cde1b81a6fe74d8
Author: Steven Webster <email address hidden>
Date: Wed Apr 26 12:10:23 2017 -0400

    Improve comment for PCI port binding update

    This commit expands the comment block for PCI logic in
    _update_port_binding_for_instance() to explain the cases
    where a migration may or may not be present.

    Change-Id: I1e699367576fbabe78fae0949588b3f40fe08da4
    Related-Bug: #1677621

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/484381

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484381
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b930336854bffec1bb81b6d67079a4df59e0af19
Submitter: Zuul
Branch: master

commit b930336854bffec1bb81b6d67079a4df59e0af19
Author: Steven Webster <email address hidden>
Date: Mon Jun 12 17:10:03 2017 -0400

    Fix instance evacuation with PCI devices

    update_port_binding_for_instance() now checks that a valid migration
    object exists as a parameter before any mapping between old/new PCI
    devices can occur. A migration should be present in the case of a
    cold migration, resize, or evacuation.

    An evacuation (being a special case of a rebuild) however, will not
    pass a migration to update_port_binding_for_instance, as it
    is called directly from setup_instance_network(). This calling function
    does not currently take a migration parameter, even though one will
    certainly exist for an evacuation.

    This commit adds an optional migration parameter to
    setup_instance_network_on_host() and passes any migration object to
    the port update routine.

    Closes-Bug: #1703629
    Related-Bug: #1677621
    Related-Bug: #1630698

    Change-Id: I4e394c8d275995eac4b049a7b1329ea90f2394be

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/590059

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/590062

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/590070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/382853
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8e052c7fe9262c38da9d8b5f9a0ee889d9c1c6be
Submitter: Zuul
Branch: master

commit 8e052c7fe9262c38da9d8b5f9a0ee889d9c1c6be
Author: paul-carlton2 <email address hidden>
Date: Thu Oct 6 12:02:15 2016 +0100

    Update nova network info when doing rebuild for evacuate operation

    When nova evacuate or host-evacuate are used to recreate instances with
    sriov ports the instances are allocated new device ids on the target and
    neutron is updated accordingly. However the network info data passed
    to the driver spawn method is not updated and thus the instance tries
    to use the device id they were allocated on the source node. If a pre
    existing instance is using that device id or no such device exists on
    the target node then the instance will fail to start.

    Co-Authored-By: Steven Webster <email address hidden>
    Change-Id: I860ab9cf3f9a38bd4ea5bceecda8105b6fee93dc
    Closes-Bug: #1630698
    Related-Bug: #1677621

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590062
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=559de0d0a3df92ef9dca21edfaa754a2848013c3
Submitter: Zuul
Branch: stable/queens

commit 559de0d0a3df92ef9dca21edfaa754a2848013c3
Author: paul-carlton2 <email address hidden>
Date: Thu Oct 6 12:02:15 2016 +0100

    Update nova network info when doing rebuild for evacuate operation

    When nova evacuate or host-evacuate are used to recreate instances with
    sriov ports the instances are allocated new device ids on the target and
    neutron is updated accordingly. However the network info data passed
    to the driver spawn method is not updated and thus the instance tries
    to use the device id they were allocated on the source node. If a pre
    existing instance is using that device id or no such device exists on
    the target node then the instance will fail to start.

    NOTE(mriedem): The test had to be modified in the backport because
    notify_about_instance_rebuild and _check_trusted_certs didn't exist
    in Queens.

    Co-Authored-By: Steven Webster <email address hidden>
    Change-Id: I860ab9cf3f9a38bd4ea5bceecda8105b6fee93dc
    Closes-Bug: #1630698
    Related-Bug: #1677621
    (cherry picked from commit 8e052c7fe9262c38da9d8b5f9a0ee889d9c1c6be)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/605881

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/605882

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590059
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9c1a58dd033fa8feeb1175956d57dc90aa55acd
Submitter: Zuul
Branch: stable/pike

commit b9c1a58dd033fa8feeb1175956d57dc90aa55acd
Author: Steven Webster <email address hidden>
Date: Mon Jun 12 17:10:03 2017 -0400

    Fix instance evacuation with PCI devices

    update_port_binding_for_instance() now checks that a valid migration
    object exists as a parameter before any mapping between old/new PCI
    devices can occur. A migration should be present in the case of a
    cold migration, resize, or evacuation.

    An evacuation (being a special case of a rebuild) however, will not
    pass a migration to update_port_binding_for_instance, as it
    is called directly from setup_instance_network(). This calling function
    does not currently take a migration parameter, even though one will
    certainly exist for an evacuation.

    This commit adds an optional migration parameter to
    setup_instance_network_on_host() and passes any migration object to
    the port update routine.

    Closes-Bug: #1703629
    Related-Bug: #1677621
    Related-Bug: #1630698

    Change-Id: I4e394c8d275995eac4b049a7b1329ea90f2394be
    (cherry picked from commit b930336854bffec1bb81b6d67079a4df59e0af19)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/590070
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c5d8594cb62c1397183888133bef68fe28c62c7a
Submitter: Zuul
Branch: stable/pike

commit c5d8594cb62c1397183888133bef68fe28c62c7a
Author: paul-carlton2 <email address hidden>
Date: Thu Oct 6 12:02:15 2016 +0100

    Update nova network info when doing rebuild for evacuate operation

    When nova evacuate or host-evacuate are used to recreate instances with
    sriov ports the instances are allocated new device ids on the target and
    neutron is updated accordingly. However the network info data passed
    to the driver spawn method is not updated and thus the instance tries
    to use the device id they were allocated on the source node. If a pre
    existing instance is using that device id or no such device exists on
    the target node then the instance will fail to start.

    Conflicts:
          nova/compute/manager.py
          nova/tests/unit/compute/test_compute_mgr.py

    NOTE(mriedem): The conflicts are due to change
    I00eab47edf1150788777300680e853a872c1db40 and change
    I752617066bb2167b49239ab9d17b0c89754a3e12 not being in Pike.

    Co-Authored-By: Steven Webster <email address hidden>
    Change-Id: I860ab9cf3f9a38bd4ea5bceecda8105b6fee93dc
    Closes-Bug: #1630698
    Related-Bug: #1677621
    (cherry picked from commit 8e052c7fe9262c38da9d8b5f9a0ee889d9c1c6be)
    (cherry picked from commit 559de0d0a3df92ef9dca21edfaa754a2848013c3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605881
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8b3b8e5657ffe0f12e1ccfbe6d978e62a2bfdc89
Submitter: Zuul
Branch: stable/ocata

commit 8b3b8e5657ffe0f12e1ccfbe6d978e62a2bfdc89
Author: Steven Webster <email address hidden>
Date: Mon Jun 12 17:10:03 2017 -0400

    Fix instance evacuation with PCI devices

    update_port_binding_for_instance() now checks that a valid migration
    object exists as a parameter before any mapping between old/new PCI
    devices can occur. A migration should be present in the case of a
    cold migration, resize, or evacuation.

    An evacuation (being a special case of a rebuild) however, will not
    pass a migration to update_port_binding_for_instance, as it
    is called directly from setup_instance_network(). This calling function
    does not currently take a migration parameter, even though one will
    certainly exist for an evacuation.

    This commit adds an optional migration parameter to
    setup_instance_network_on_host() and passes any migration object to
    the port update routine.

    Closes-Bug: #1703629
    Related-Bug: #1677621
    Related-Bug: #1630698

    Change-Id: I4e394c8d275995eac4b049a7b1329ea90f2394be
    (cherry picked from commit b930336854bffec1bb81b6d67079a4df59e0af19)
    (cherry picked from commit b9c1a58dd033fa8feeb1175956d57dc90aa55acd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605882
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9e146072b6ab5c614cc4502170cc686cc8cc7bce
Submitter: Zuul
Branch: stable/ocata

commit 9e146072b6ab5c614cc4502170cc686cc8cc7bce
Author: paul-carlton2 <email address hidden>
Date: Thu Oct 6 12:02:15 2016 +0100

    Update nova network info when doing rebuild for evacuate operation

    When nova evacuate or host-evacuate are used to recreate instances with
    sriov ports the instances are allocated new device ids on the target and
    neutron is updated accordingly. However the network info data passed
    to the driver spawn method is not updated and thus the instance tries
    to use the device id they were allocated on the source node. If a pre
    existing instance is using that device id or no such device exists on
    the target node then the instance will fail to start.

    Conflicts:
          nova/compute/manager.py
          nova/tests/unit/compute/test_compute_mgr.py

    NOTE(mriedem): The conflict is due to not having change
    Iddae8074554995df22b656bb2e9bddaec6d775cc in Ocata.

    Co-Authored-By: Steven Webster <email address hidden>
    Change-Id: I860ab9cf3f9a38bd4ea5bceecda8105b6fee93dc
    Closes-Bug: #1630698
    Related-Bug: #1677621
    (cherry picked from commit 8e052c7fe9262c38da9d8b5f9a0ee889d9c1c6be)
    (cherry picked from commit 559de0d0a3df92ef9dca21edfaa754a2848013c3)
    (cherry picked from commit c5d8594cb62c1397183888133bef68fe28c62c7a)