[RFE] Admin extension for viewing & remediating neutron/backend resource mappings

Bug #1563538 reported by Boden R
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

[Overview + Motivation]
Cloud software is complex. Resources abstracted at the Cloud layer must be realized on backend system(s) (native Operating System or other potentially distributed service(s)) and an association between the Cloud and backend resource must be maintained and kept in sync with each other. Regardless of how well we code and/or test our systems, sometimes unforeseen circumstances occur that cause the resource associations to become out of sync.

Often times one or more resources in the association are deleted and others are left behind. For example, connectivity errors while a delete operation is running in the Cloud cause the backend resource to not be (fully) cleaned up. This situation is often referred to as "orphaned objects / resources". Remediation of such circumstances not only requires the technical "know how", but also orphaned resource identification. Failure to remediate leads to resource leakage among other potential problems within the Cloud and/or backend systems.

[Proposed Solution]
The Pluggable Backend Association Mapping (PBAM) extension for neutron aims to address these issues by giving OpenStack neutron administrators additional visibility and remediation capabilities across heterogeneous (services, plugins, etc.) neutron backend resource mappings. Using the unified consistent northbound REST API provided by the extension, admins can quickly inventory and identify orphaned resources within neutron. They can then use the same consistent API to clean-up or re-create those orphaned backend resources as necessary.

Today, we see OpenStack consumers writing their own orphaned resource clean-up scripts and/or struggling to ensure resource mappings are in sync. What we propose here is a neutron core extension that automates this task, easing the administrative burden that exists today.

[1] PBAM PoC including DHCP Agent mappings reference implementation and python-neutronclient bindings and documentation on the PoC, it's API, etc.:

[2] Video demo:

Tags: rfe
Boden R (boden)
description: updated
Revision history for this message
Assaf Muller (amuller) wrote :

It seems like there's a lot of overlap with another proposal:

Revision history for this message
Boden R (boden) wrote :

@amuller thanks for the ref.

I added some notes to [1] in regards to how these *might* fit together.

[1] https://etherpad.openstack.org/p/neutron-troubleshooting

Revision history for this message
Ash Young (ashleeyoung) wrote :

I definitely see synergy between the two. What I'm curious about is how we can make this also more aware of supporting services, as perhaps a type of resources. Maybe a resource is unavailable, but another with alternative capabilities, might be available. Just a thought.

Revision history for this message
Boden R (boden) wrote :

@ashlee-i thanks for the input...

If you don't mind; could you expand on your thought a little bit more? Perhaps a use case or workflow for what you envisioned?

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

Let's keep the discussion in one place, otherwise these types of initiatives will have the opposite effects of allowing us to get to any form or consensus.

The demo looks cool though. I haven't been able to assess how much of a need to tightly integrate with Neutron there is, but if there is none, then kudos to the effort!

Changed in neutron:
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron-specs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/308973

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron-specs (master)

Reviewed: https://review.openstack.org/308973
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=dc11da5109759d13636aaaef35420fa4ac1d88d6
Submitter: Jenkins
Branch: master

commit dc11da5109759d13636aaaef35420fa4ac1d88d6
Author: Boden R <email address hidden>
Date: Wed Feb 15 15:47:07 2017 -0700

    Neutron resource diagnostics

    This spec proposes the introduction of a neutron diagnostics framework
    and API extension capable collecting resource diagnostics across
    neutron API and agent nodes. To keep the spec containable, the proposal
    suggests only providing a sample diagnostic check and reiterating on
    concrete diagnostics once we get the plumbing in place.

    While this spec has some inspiration from nova diagnostics [1],
    the approach herein is more generic and extensible supporting a
    broader set of use cases longer term.

    Finally it seeks to pave the way for supporting use case / features
    proposed in the related bugs.

    [1] https://wiki.openstack.org/wiki/Nova_VM_Diagnostics

    Related-Bug: #1507499
    Related-Bug: #1519537
    Related-Bug: #1537686
    Related-Bug: #1563538

    Change-Id: Id534acb1593f1fe210c561b1451656dce69514db

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.