ha interface cleanup can leave orphans

Bug #1540271 reported by Kevin Benton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Kevin Benton

Bug Description

The HA delete_router routine may leave behind stranded interfaces if it fails to delete the HA ports because it deletes the router object before removing the ports. This failure could be an issue in the loaded ML2 drivers, or it could simply be that the neutron server process was killed before completing the deletion.

These orphaned interfaces are then difficult to delete via the API due to their device_owner field.

https://github.com/openstack/neutron/blob/6bcacc114281b1017acae7e436a56b2da09f69f1/neutron/db/l3_hamode_db.py#L504-L512

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/274570

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/274571

Assaf Muller (amuller)
tags: added: l3-ha
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/274570
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3b41808b8624f5442d65ba638d1a2c1bdc22be00
Submitter: Jenkins
Branch: master

commit 3b41808b8624f5442d65ba638d1a2c1bdc22be00
Author: Kevin Benton <email address hidden>
Date: Mon Feb 1 01:17:22 2016 -0800

    Only prevent l3 port deletion if router exists

    This patch modifies the prevent_l3_port_deletion method to
    actually look up the router_id in the device_owner field to
    confirm that the router exists before claiming the port is
    in use. This will allow users to delete ports that may have
    been orphaned due to race conditions in the cleanup of router
    interfaces.

    Partial-Bug: #1540271
    Change-Id: Ieffe632f3f3098baf202d3795ab5182982e234bd

Changed in neutron:
importance: Undecided → Medium
Changed in neutron:
assignee: Kevin Benton (kevinbenton) → Ann Kamyshnikova (akamyshnikova)
Changed in neutron:
assignee: Ann Kamyshnikova (akamyshnikova) → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/308142

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/308142
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6b33c51a32e075c57630160b5b0544ad29f1a3bb
Submitter: Jenkins
Branch: stable/liberty

commit 6b33c51a32e075c57630160b5b0544ad29f1a3bb
Author: Kevin Benton <email address hidden>
Date: Mon Feb 1 01:17:22 2016 -0800

    Only prevent l3 port deletion if router exists

    This patch modifies the prevent_l3_port_deletion method to
    actually look up the router_id in the device_owner field to
    confirm that the router exists before claiming the port is
    in use. This will allow users to delete ports that may have
    been orphaned due to race conditions in the cleanup of router
    interfaces.

    Conflicts:
     neutron/tests/unit/db/test_l3_db.py

    Closes-Bug: #1566678
    Partial-Bug: #1540271
    Change-Id: Ieffe632f3f3098baf202d3795ab5182982e234bd
    (cherry picked from commit 3b41808b8624f5442d65ba638d1a2c1bdc22be00)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/343936

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/274571
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0074b943d6efc5526c687cc9ca3857aabc2dbc63
Submitter: Jenkins
Branch: master

commit 0074b943d6efc5526c687cc9ca3857aabc2dbc63
Author: Kevin Benton <email address hidden>
Date: Sat Apr 9 12:15:56 2016 -0700

    Add RouterPort binding to ha interface creation

    The L3 HA interface cleanup routines during router deletion were
    occuring after the parent router resource was deleted. This meant
    that if they failed to delete, they would be orphaned since the
    parent router was already gone.

    This order of router delete and then HA delete was intentional
    because deleting them before calling the parent router delete
    function would cause them to be deleted before the checks to see
    if the router was in use. If the router was in use, the HA interface
    would be removed and would leave an HA router in an inconsitent state.
    (See I956d0094ae6e2412e859d79feeb4003941d2bb4b for details)

    This patch just has the L3 HA code add a RouterPort binding for the
    HA interfaces. This allows the parent router_delete call to delete the
    HA interfaces automatically after doing the in-use check but before
    deleting the actual router.

    In order to avoid a migration in a back-port, this is compatible with
    previously created routers as well by continuing to call
    _delete_ha_interfaces after the parent router is deleted. This means
    that any routers created before this patch will potentially leave
    orphans that will have to be manually deleted via the port API.

    Closes-Bug: #1540271
    Change-Id: Ifd3e007aaf2a2ed8123275aa3a9f540838e3c003

Changed in neutron:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/343936
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=91614d33c101b72c0fab965bf58dddde2fa25c87
Submitter: Jenkins
Branch: master

commit 91614d33c101b72c0fab965bf58dddde2fa25c87
Author: Kevin Benton <email address hidden>
Date: Thu Jul 14 03:22:02 2016 -0700

    Add RouterPort bindings for all HA ports

    Adding these bindings allows us to remove the special-cased HA
    interface deletion in l3_hamode_db that had the potential for
    leaving orphaned ports on failures. With the bindings present,
    the interface deletion is handled before the router delete so
    a failure to delete the interface will prevent router deletion.

    Related-Bug: #1540271
    Change-Id: I2de8503742661c18a2ec2c5ade7ec58ea380e749

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0b3

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.