deleting an interface hits a race condition and fails

Bug #1297992 reported by Joe Gordon
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
New
Undecided
Unassigned
neutron
New
Undecided
Aaron Rosen

Bug Description

We are seeing nova REST calls to delete an attached interface fail to work:
  DELETE http://127.0.0.1:8774/v3/servers/UUID/os-attach-interfaces/UUID
  DELETE http://127.0.0.1:8774/v2/UUID/servers/UUID/os-interface/UUID

In nova-cpu we see

DEBUG neutronclient.client [-] RESP:{'status': '200', 'content-length': '13', 'content-location': 'http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp', 'date': 'Wed, 26 Mar 2014 00:44:04 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-59333b8f-072d-4bce-b5cc-03d9af48dcc8'} {"ports": []}
/python-neutronclient/neutronclient/common/utils.py:179
 nova.network.api [req-61feb3de-431b-4dcc-81c7-8a8e5b6b03ed AttachInterfacesV3Test-673399961 AttachInterfacesV3Test-295185211] Updating cache with info: [VIF({'ovs_interfaceid': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.4'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap530319a7-b4', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:e8:72:f4', 'active': True, 'type': u'ovs', 'id': u'530319a7-b4d7-4289-80f8-2a1ec39c642a', 'qbg_params': None}), VIF({'ovs_interfaceid': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 'address': u'10.100.0.2'})], 'version': 4, 'meta': {}, 'dns': [], 'routes': [], 'cidr': u'10.100.0.0/28', 'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': u'10.100.0.1'})})], 'meta': {'injected': False, 'tenant_id': u'e9a96753b0004178bf8dd2007b01d55f'}, 'id': u'0c95c39b-611f-4880-9e56-cae0eb948225', 'label': u'AttachInterfacesV3Test-1465112406-network'}), 'devname': u'tap8077e5f2-e6', 'qbh_params': None, 'meta': {}, 'address': u'fa:16:3e:83:34:a9', 'active': True, 'type': u'ovs', 'id': u'8077e5f2-e683-42f4-b9df-9c049be25cad', 'qbg_params': None})] update_instance_cache_with_nw_info /opt/stack/new/nova/nova/network/api.py:74
ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
TRACE oslo.messaging.rpc.dispatcher incoming.message))
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface
TRACE oslo.messaging.rpc.dispatcher "attached") % port_id)
TRACE oslo.messaging.rpc.dispatcher PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached
TRACE oslo.messaging.rpc.dispatcher
ERROR oslo.messaging._drivers.common [-] Returning exception Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached to caller
ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/new/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/new/nova/nova/compute/manager.py", line 4370, in detach_interface\n "attached") % port_id)\n', 'PortNotFound: Port 760ab1e4-10ae-4170-b491-54ec71efd952 is not attached\n']

http://logs.openstack.org/22/82922/2/gate/gate-tempest-dsvm-neutron/87c67c5/logs/screen-n-cpu.txt.gz#_2014-03-26_00_44_04_250

It looks like when nova does 'curl -i http://127.0.0.1:9696/v2.0/ports.json?network_id=0c95c39b-611f-4880-9e56-cae0eb948225&device_owner=network%3Adhcp' neutron is returning an empty list.

logstash query: message:"PortNotFound: Port" AND message:"is not attached" AND filename:logs*screen-n-cpu.txt

Aaron Rosen (arosen)
Changed in neutron:
assignee: nobody → Aaron Rosen (arosen)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.