Remaining ports are not unbound if one port is missing

Bug #1974173 reported by Stephen Finucane
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Stephen Finucane
Train
New
Undecided
Unassigned
Ussuri
New
Undecided
Unassigned
Victoria
New
Undecided
Unassigned
Wallaby
New
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned

Bug Description

As part of the instance deletion process, we must unbind ports associated with said instance. To do this, we loop over all ports currently attached to an instance. However, if neutron returns HTTP 404 (Not Found) for any of these ports, we will return early and fail to unbind the remaining ports. We've seen the problem in the context of Kubernetes on OpenStack. Our deinstaller is brute-force, so it deletes ports and servers at the same time, so a race means the port can get deleted early. This normally wouldn't be an issue as we'd just "untrunk" it and proceed to delete it. But that won't work for SR-IOV ports as in that case you cannot "untrunk" bound ports.

The solution here is obvious: if we fail to find a port, we should simply skip that and continue unbinding everything else.

Tags: neutron
tags: added: neutron
Changed in nova:
assignee: nobody → Stephen Finucane (stephenfinucane)
importance: Undecided → Medium
status: New → Confirmed
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/842528
Committed: https://opendev.org/openstack/nova/commit/9e0dcb52ab308a63c6a18e47d1850cc3ade4d807
Submitter: "Zuul (22348)"
Branch: master

commit 9e0dcb52ab308a63c6a18e47d1850cc3ade4d807
Author: Stephen Finucane <email address hidden>
Date: Thu May 19 12:04:08 2022 +0100

    neutron: Unbind remaining ports after PortNotFound

    Just because we encountered a PortNotFound error when unbinding a port
    doesn't mean we should stop unbinding the remaining ports. If this error
    is encountered, simply continue with the other ports.

    While we're here, we clean up some other tests related to '_unbind_port'
    since they're clearly duplicates.

    Change-Id: Id04e0df12829df4d8929e03a8b76b5cbe0549059
    Signed-off-by: Stephen Finucane <email address hidden>
    Closes-Bug: #1974173

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/842584

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/842586

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/842587

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/842588

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/842590

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/842592

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/nova/+/842584
Committed: https://opendev.org/openstack/nova/commit/6f32b118640ee466b58155b5ecd50e041b4a2e7e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 6f32b118640ee466b58155b5ecd50e041b4a2e7e
Author: Stephen Finucane <email address hidden>
Date: Thu May 19 12:04:08 2022 +0100

    neutron: Unbind remaining ports after PortNotFound

    Just because we encountered a PortNotFound error when unbinding a port
    doesn't mean we should stop unbinding the remaining ports. If this error
    is encountered, simply continue with the other ports.

    While we're here, we clean up some other tests related to '_unbind_port'
    since they're clearly duplicates.

    Change-Id: Id04e0df12829df4d8929e03a8b76b5cbe0549059
    Signed-off-by: Stephen Finucane <email address hidden>
    Closes-Bug: #1974173
    (cherry picked from commit 9e0dcb52ab308a63c6a18e47d1850cc3ade4d807)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 26.0.0.0rc1

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/842586
Committed: https://opendev.org/openstack/nova/commit/d322f8e8b5a95fd60dab9c6c543b4c77ea893836
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d322f8e8b5a95fd60dab9c6c543b4c77ea893836
Author: Stephen Finucane <email address hidden>
Date: Thu May 19 12:04:08 2022 +0100

    neutron: Unbind remaining ports after PortNotFound

    Just because we encountered a PortNotFound error when unbinding a port
    doesn't mean we should stop unbinding the remaining ports. If this error
    is encountered, simply continue with the other ports.

    While we're here, we clean up some other tests related to '_unbind_port'
    since they're clearly duplicates.

    Conflicts:
      nova/tests/unit/network/test_neutron.py

    NOTE(stephenfin): Conflicts are due to the absence of change
    I8058902df167239fa455396d3595a56bcf472b2b ("neutron: Rework how we check
    for extensions") which we don't want to backport.

    Change-Id: Id04e0df12829df4d8929e03a8b76b5cbe0549059
    Signed-off-by: Stephen Finucane <email address hidden>
    Closes-Bug: #1974173
    (cherry picked from commit 9e0dcb52ab308a63c6a18e47d1850cc3ade4d807)
    (cherry picked from commit 6f32b118640ee466b58155b5ecd50e041b4a2e7e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.2.0

This issue was fixed in the openstack/nova 24.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 25.1.0

This issue was fixed in the openstack/nova 25.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/train)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/842592
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/ussuri)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/842590
Reason: stable/ussuri branch of openstack/nova transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/victoria)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/842588
Reason: stable/victoria branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/wallaby)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/842587
Reason: stable/wallaby branch of openstack/nova is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.