Rally test boot_and_delete_server_with_secgroups fails

[Gary Kotton]

When running rally scenarios the test fails with PortNotFoundClient exception.
From the logs we see the following:

Failure:
-------

▼ 11 GetResourceFailure Failed to get the resource <Server: s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a9471015-e72e-42cf-acc3-8b361415d91e)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 71, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/security_group.py", line 167, in run
    self._delete_server(server)
  File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 451, in _delete_server
    check_interval=CONF.openstack.nova_server_delete_poll_interval
  File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 214, in wait_for_status
    resource = update_resource(resource)
  File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line 80, in _get_from_manager
    raise exceptions.GetResourceFailure(resource=resource, err=e)
GetResourceFailure: Failed to get the resource <Server: s_rally_41596539_fMbobPdS>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a9471015-e72e-42cf-acc3-8b361415d91e)

Neutron:
-------
port-creation:
2018-04-22 09:18:31.984 32139 DEBUG vmware_nsxlib.v3.client [req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] REST call: POST api/v1/logical-ports. Headers: {'X-NSX-EUSER': u'f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default', 'Content-Type': 'application/json', 'X-NSX-EREQID': 'req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b', 'Accept': 'application/json'}. Body: {"address_bindings": [{"ip_address": "100.1.124.13", "mac_address": "fa:16:3e:d4:3a:8f"}], "admin_state": "UP", "attachment": null, "description": "", "display_name": "", "logical_switch_id": "dfc6fa69-6df7-44db-ba43-68152104103f", "switching_profile_ids": [{"key": "SpoofGuardSwitchingProfile", "value": "0bb5dd95-1e3d-464e-a9d5-bd040bd91d9d"}], "tags": [{"scope": "os-neutron-port-id", "tag": "1954b052-bc2c-48ba-aa24-e8c96fac1723"}, {"scope": "os-project-id", "tag": "c87b5a2bc4e649069e9cfa4bbc874663"}, {"scope": "os-project-name", "tag": "c_rally_41596539_tkcSTCuv"}, {"scope": "os-api-version", "tag": "12.0.0.dev8294691"}, {"scope": "os-security-group", "tag": "a4d155d5-285a-4e9a-8982-04f6a242d3ed"}, {"scope": "os-security-group", "tag": "OS-Default-Section"}]} _rest_call /usr/lib/python2.7/dist-packages/vmware_nsxlib/v3/client.py:201
.
.
2018-04-22 09:18:32.278 32139 INFO neutron.wsgi [req-7f2bece2-7bd3-4e25-a5be-9c363e77bd5b f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.239,10.127.108.237 "POST /v2.0/ports HTTP/1.1" status: 201 len: 983 time: 2.8035491

port-deletion:

2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-aa24-e8c96fac1723 HTTP/1.1" status: 204 len: 168 time: 2.4448841

Nova:
----
2018-04-22 09:18:55.539 19221 DEBUG oslo_concurrency.lockutils [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Lock "9ab78fdd-5798-4426-baf2-2706d0d69345" acqu
ired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:276
2018-04-22 09:18:55.541 19221 DEBUG oslo_concurrency.lockutils [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Lock "9ab78fdd-5798-4426-baf2-2706d0d69345" rele
ased by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.002s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:288
2018-04-22 09:18:55.662 19221 DEBUG nova.policy [req-824643af-4107-46a9-ba5a-950fcec27ba4 8d478ca1fa274601a50f8d2a38c5e70f 69fad6d3240645b5854b52bca7e9b85e - default default] Policy check for os_compute_api:os-extended-server-attributes f
ailed with credentials {'service_roles': [], 'user_id': u'8d478ca1fa274601a50f8d2a38c5e70f', 'roles': [u'_member_'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'servi
ce_project_domain_id': None, 'is_admin_project': False, 'is_admin': False, 'project_id': u'69fad6d3240645b5854b52bca7e9b85e', 'project_domain_id': u'default'} authorize /usr/lib/python2.7/dist-packages/nova/policy.py:168
2018-04-22 09:18:55.774 19225 DEBUG neutronclient.v2_0.client [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Error message: {"NeutronError": {"message": "Port
 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.", "type": "PortNotFound", "detail": ""}} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Caught error: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723
could not be found.
Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']: PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Traceback (most recent call last):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 82, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application, catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 92, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self._log_req(req, res, start)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self.force_reraise()
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/requestlog.py", line 87, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res = req.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application, catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 112, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return request.get_response(self.application)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application, catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response = req.get_response(self._app)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1316, in send
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack application, catch_exc_info=False)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1280, in call_application
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack response = self.app(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return resp(environ, start_response)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 580, in __call__
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack content_type, body, accept)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 654, in _process_stack
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack request, action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 526, in process_extensions
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **action_args)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 534, in show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack return self._show(req, resp_obj)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 530, in _show
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self._extend_servers(req, [resp_obj.obj['server']])
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/security_groups.py", line 512, in _extend_servers
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack servers))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 376, in get_instances_security_groups_bindings
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 335, in _get_ports_from_server_list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports'))
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack **_params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in list
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack for r in self._pagination(collection, path, **params):
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack res = self.get(path, params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers, params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack headers=headers, params=params)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 294, in do_request
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack self._handle_fault_response(status_code, replybody, resp)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack ret = obj(*args, **kwargs)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack exception_handler_v20(status_code, error_body)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack request_ids=request_ids)
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50 returned with HTTP 500
2018-04-22 09:18:55.822 19225 DEBUG nova.api.openstack.wsgi [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1064

At a deeper looks we see:
After looking more into the logs and the flow, here are the relevant log lines that explain the issue, and indicate this is an exception not being handled in Nova (see here https://github.com/openstack/nova/blob/master/nova/network/security_group/neutron_driver.py#L378), note the timestamps:

Port ID: 1954b052-bc2c-48ba-aa24-e8c96fac1723
Nova instance UUID: 3e714b48-5193-4b13-b6c9-3374488b2e50

Neutron logs
=============
Port deletion:
2018-04-22 09:18:55.552 32135 INFO neutron.wsgi [req-774309da-6fc6-4521-aed2-37f2eb01b5b4 f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.239,10.127.108.237 "DELETE /v2.0/ports/1954b052-bc2c-48ba-aa24-e8c96fac1723 HTTP/1.1" status: 204 len: 168 time: 2.4448841

GET ports failure:
2018-04-22 09:18:55.770 31303 INFO neutron.wsgi [req-35436010-4faa-42de-ab84-a783f48b4f92 f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] 10.127.108.236,10.127.108.237 "GET /v2.0/ports?device_id=3e714b48-5193-4b13-b6c9-3374488b2e50 HTTP/1.1" status: 404 len: 333 time: 0.5505300

Nova log
========
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack PortNotFoundClient: Port 1954b052-bc2c-48ba-aa24-e8c96fac1723 could not be found.
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack Neutron server returns request_ids: ['req-35436010-4faa-42de-ab84-a783f48b4f92']
2018-04-22 09:18:55.778 19225 ERROR nova.api.openstack
2018-04-22 09:18:55.819 19225 INFO nova.api.openstack [req-a9471015-e72e-42cf-acc3-8b361415d91e f8e81abca4094924831b9538389b8d5b c87b5a2bc4e649069e9cfa4bbc874663 - default default] http://192.168.20.101:8774/v2.1/c87b5a2bc4e649069e9cfa4bbc874663/servers/3e714b48-5193-4b13-b6c9-3374488b2e50 returned with HTTP 500

If there is a exception in self._get_secgroups_from_port_list should the security groups returned be []? Or a better exception raised. This is due to a race in the request in the retrieing the security groups and the deletion of the instance. The instance does not exist at this stage...

The problem is discovered in stable queens

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/565183

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/565183
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a0179f7ab9b9c7f07ce9c4dd681486de5a4443a
Submitter: Zuul
Branch: master

commit 1a0179f7ab9b9c7f07ce9c4dd681486de5a4443a
Author: Radoslav Gerganov <email address hidden>
Date: Mon Apr 30 13:18:26 2018 +0300

    Handle PortNotFoundClient exception when getting ports

    There could be a race between deleting an instance and retrieving its
    port groups from Neutron. In this case PortNotFoundClient is raised and
    it can be safely ignored.

    Change-Id: I31c9ea8628c6f3985f8e9118d9687bbfb8789b68
    Closes-Bug: #1767058

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/567211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567211
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3bf3986c14ed14b9f460161aa4cc2950da5f56f5
Submitter: Zuul
Branch: stable/queens

commit 3bf3986c14ed14b9f460161aa4cc2950da5f56f5
Author: Radoslav Gerganov <email address hidden>
Date: Mon Apr 30 13:18:26 2018 +0300

    Handle PortNotFoundClient exception when getting ports

    There could be a race between deleting an instance and retrieving its
    port groups from Neutron. In this case PortNotFoundClient is raised and
    it can be safely ignored.

    Change-Id: I31c9ea8628c6f3985f8e9118d9687bbfb8789b68
    Closes-Bug: #1767058
    (cherry picked from commit 1a0179f7ab9b9c7f07ce9c4dd681486de5a4443a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b2 development milestone.