Concurrent calls to DELETE os-floating-ips can raise uncaught neutronclient.common.exceptions.NotFound

Bug #1649852 reported by Guillaume Espanel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Stephen Finucane
Ocata
Fix Committed
Medium
Matt Riedemann

Bug Description

Description
============

Concurrent calls to DELETE os-floating-ips/floating-ip-id can cause neutronclient to raise
a neutronclient.common.exceptions.NotFound exception uncaught by Nova which in turn returns
a 500 Error.

Steps to reproduce
===================

Tested on a mitaka devstack :

 - source openrc
 - run this humble script :

FIP_ID=`nova floating-ip-create | grep public | awk '{print($2)}'`
TENANT_ID=`keystone token-get | grep " tenant_id " | awk '{print($4)}'`
TOKEN_ID=`keystone token-get | grep " id " | awk '{print($4)}'`

curl -g -i -X DELETE http://127.0.0.1:8774/v2.1/$TENANT_ID/os-floating-ips/$FIP_ID -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN_ID"&
curl -g -i -X DELETE http://127.0.0.1:8774/v2.1/$TENANT_ID/os-floating-ips/$FIP_ID -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN_ID"&
curl -g -i -X DELETE http://127.0.0.1:8774/v2.1/$TENANT_ID/os-floating-ips/$FIP_ID -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN_ID"&
curl -g -i -X DELETE http://127.0.0.1:8774/v2.1/$TENANT_ID/os-floating-ips/$FIP_ID -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN_ID"

Expected result
================

First call to go trough should return a 202, and all other calls should probably return 404 or 409 if deletion is in progress but not 500.

Actual result
==============

Some calls to DELETE get a 500 Error response.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/410815

Changed in nova:
assignee: nobody → Guillaume Espanel (guillaume-espanel)
status: New → In Progress
Changed in nova:
assignee: Guillaume Espanel (guillaume-espanel) → Stephen Finucane (stephenfinucane)
Changed in nova:
assignee: Stephen Finucane (stephenfinucane) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Guillaume Espanel (guillaume-espanel)
importance: Undecided → Medium
Changed in nova:
assignee: Guillaume Espanel (guillaume-espanel) → Stephen Finucane (stephenfinucane)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/410815
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d99197aece6451013d1de1f08c1af16832ee0e7e
Submitter: Jenkins
Branch: master

commit d99197aece6451013d1de1f08c1af16832ee0e7e
Author: Guillaume Espanel <email address hidden>
Date: Wed Dec 14 15:29:02 2016 +0100

    Catch neutronclient.NotFound on floating deletion

    In some cases, trying to delete a floating IP multiple times in a short
    delay can trigger an exception beacause the floating ip deletion
    operation is not atomic. If neutronclient's call to delete fails with a
    NotFound error, we raise a 404 error to nova's client instead of a 500.

    Change-Id: I49ea7e52073148457e794d641ed17d4ef58616f8
    Co-Authored-By: Stephen Finucane <email address hidden>
    Closes-Bug: #1649852

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/471510

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/471511

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b2

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/471510
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1052ad4d5adbb2cb2794f266f21961340c534b69
Submitter: Jenkins
Branch: stable/ocata

commit 1052ad4d5adbb2cb2794f266f21961340c534b69
Author: Guillaume Espanel <email address hidden>
Date: Wed Dec 14 15:29:02 2016 +0100

    Catch neutronclient.NotFound on floating deletion

    In some cases, trying to delete a floating IP multiple times in a short
    delay can trigger an exception beacause the floating ip deletion
    operation is not atomic. If neutronclient's call to delete fails with a
    NotFound error, we raise a 404 error to nova's client instead of a 500.

    Change-Id: I49ea7e52073148457e794d641ed17d4ef58616f8
    Co-Authored-By: Stephen Finucane <email address hidden>
    Closes-Bug: #1649852
    (cherry picked from commit d99197aece6451013d1de1f08c1af16832ee0e7e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.6

This issue was fixed in the openstack/nova 15.0.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/471511
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f5e86d386485e102b7430b332e79c6155adf6528
Submitter: Jenkins
Branch: stable/newton

commit f5e86d386485e102b7430b332e79c6155adf6528
Author: Guillaume Espanel <email address hidden>
Date: Wed Dec 14 15:29:02 2016 +0100

    Catch neutronclient.NotFound on floating deletion

    In some cases, trying to delete a floating IP multiple times in a short
    delay can trigger an exception beacause the floating ip deletion
    operation is not atomic. If neutronclient's call to delete fails with a
    NotFound error, we raise a 404 error to nova's client instead of a 500.

    Change-Id: I49ea7e52073148457e794d641ed17d4ef58616f8
    Co-Authored-By: Stephen Finucane <email address hidden>
    Closes-Bug: #1649852
    (cherry picked from commit d99197aece6451013d1de1f08c1af16832ee0e7e)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.8

This issue was fixed in the openstack/nova 14.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.