Neutron-API throws InternalServerError when cleaning up rally instance

[Alexander Balderson]

This happened during the Openstack charm release run on the rally test VMTasks.check_designate_dns_resolving.

During the cleanup, the nova logs show a failure to delete the port used by the instance resulting in a test failure.

2021-01-19 20:35:57.279 1409476 WARNING nova.network.neutron [-] [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] Failed to delete port f984ae1a-39e3-4392-9710-b757aba876a3 for instance.: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-1f477de2-870e-4788-9d32-0e5bddd2d692']
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] Traceback (most recent call last):
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1654, in _delete_ports
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] neutron.delete_port(port)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 184, in wrapper
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] ret = obj(*args, **kwargs)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 815, in delete_port
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] return self.delete(self.port_path % (port))
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 184, in wrapper
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] ret = obj(*args, **kwargs)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 349, in delete
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] return self.retry_request("DELETE", action, body=body,
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 184, in wrapper
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] ret = obj(*args, **kwargs)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 330, in retry_request
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] return self.do_request(method, action, body=body,
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 184, in wrapper
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] ret = obj(*args, **kwargs)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] self._handle_fault_response(status_code, replybody, resp)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 184, in wrapper
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] ret = obj(*args, **kwargs)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] exception_handler_v20(status_code, error_body)
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] raise client_exc(message=error_message,
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539] Neutron server returns request_ids: ['req-1f477de2-870e-4788-9d32-0e5bddd2d692']
2021-01-19 20:35:57.279 1409476 ERROR nova.network.neutron [instance: 64dd9c32-0eba-40f1-a83b-621ffb4da539]

the test bundle:
https://oil-jenkins.canonical.com/artifacts/d61ac279-101a-41a1-bd15-3ab84f872966/generated/generated/openstack/bundle.yaml
test crashdump:
https://oil-jenkins.canonical.com/artifacts/d61ac279-101a-41a1-bd15-3ab84f872966/generated/generated/openstack/juju-crashdump-openstack-2021-01-19-20.50.31.tar.gz
all artifacts:
https://oil-jenkins.canonical.com/artifacts/d61ac279-101a-41a1-bd15-3ab84f872966/index.html

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)