Test fails as:-
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/api/compute/servers/test_delete_server.py", line 44, in test_delete_server_while_in_building_state
self.client.delete_server(server['id'])
File "/opt/stack/tempest/tempest/lib/services/compute/servers_client.py", line 170, in delete_server
resp, body = self.delete("servers/%s" % server_id)
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/opt/stack/tempest/tempest/lib/services/compute/base_compute_client.py", line 47, in request
resp, resp_body = super(BaseComputeClient, self).request(
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
self._error_checker(resp, resp_body)
File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 922, in _error_checker
raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBReferenceError'>
Nova traces:-
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi [None req-19b3c8fc-586e-40ad-b3b7-3cc604632d69 tempest-DeleteServersTestJSON-1007295085 tempest-DeleteServersTestJSON-1007295085-project-member] Unexpected exception in API method: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: [SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, action, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: [parameters: {'created_at': datetime.datetime(2024, 5, 3, 20, 10, 51, 82097), 'updated_at': datetime.datetime(2024, 5, 3, 20, 10, 49, 930003), 'deleted_at': None, 'deleted': 0, 'action': 'delete', 'instance_uuid': '61f3c713-8ef1-4281-a3db-f199197efda3', 'request_id': 'req-19b3c8fc-586e-40ad-b3b7-3cc604632d69', 'user_id': '2aa1943eb18440ecb0ee37b064938127', 'project_id': '6d39b8d66e43454697d88a86755b45e3', 'start_time': datetime.datetime(2024, 5, 3, 20, 10, 49, 930003), 'finish_time': None, 'message': None}]
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: (Background on this error at: https://sqlalche.me/e/14/gkpj)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self.dialect.do_execute(
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi cursor.execute(statement, parameters)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result = self._query(query)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi conn.query(q)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._affected_rows = self._read_query_result(unbuffered=unbuffered)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result.read()
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi first_packet = self.connection._read_packet()
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi packet.raise_for_error()
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi raise errorclass(errno, errval)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi The above exception was the direct cause of the following exception:
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 665, in wrapped
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1102, in delete
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._delete(req.environ['nova.context'], req, id)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 905, in _delete
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self.compute_api.delete(context, instance)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 242, in inner
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 168, in inner
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2726, in delete
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._delete_instance(context, instance)
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2717, in _delete_instance
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._delete(context, instance, 'delete', self._do_delete,
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 2554, in _delete
May 03 20:10:51.134858 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._record_action_start(context, instance,
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 414, in _record_action_start
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi objects.InstanceAction.action_start(context, instance.uuid,
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/instance_action.py", line 86, in action_start
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi db_action = db.action_start(context, values)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/main/api.py", line 207, in wrapper
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/db/main/api.py", line 3900, in action_start
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi action_ref.save(context.session)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/models.py", line 38, in save
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi session.flush()
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3449, in flush
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._flush(objects)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3588, in _flush
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi with util.safe_reraise():
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi compat.raise_(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi raise exception
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi flush_context.execute()
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi rec.execute(self)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi util.preloaded.orm_persistence.save_obj(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi _emit_insert_statements(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result = connection._execute_20(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return meth(self, args_10style, kwargs_10style, execution_options)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi return connection._execute_clauseelement(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi ret = self._execute_context(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._handle_dbapi_exception(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2132, in _handle_dbapi_exception
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi util.raise_(newraise, with_traceback=exc_info[2], from_=e)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi raise exception
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self.dialect.do_execute(
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi cursor.execute(statement, parameters)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result = self._query(query)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi conn.query(q)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi self._affected_rows = self._read_query_result(unbuffered=unbuffered)
May 03 20:10:51.139041 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi result.read()
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi first_packet = self.connection._read_packet()
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi packet.raise_for_error()
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi err.raise_mysql_exception(self._data)
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi File "/opt/stack/data/venv/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi raise errorclass(errno, errval)
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell0`.`instance_actions`, CONSTRAINT `fk_instance_actions_instance_uuid` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi [SQL: INSERT INTO instance_actions (created_at, updated_at, deleted_at, deleted, action, instance_uuid, request_id, user_id, project_id, start_time, finish_time, message) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(action)s, %(instance_uuid)s, %(request_id)s, %(user_id)s, %(project_id)s, %(start_time)s, %(finish_time)s, %(message)s)]
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi [parameters: {'created_at': datetime.datetime(2024, 5, 3, 20, 10, 51, 82097), 'updated_at': datetime.datetime(2024, 5, 3, 20, 10, 49, 930003), 'deleted_at': None, 'deleted': 0, 'action': 'delete', 'instance_uuid': '61f3c713-8ef1-4281-a3db-f199197efda3', 'request_id': 'req-19b3c8fc-586e-40ad-b3b7-3cc604632d69', 'user_id': '2aa1943eb18440ecb0ee37b064938127', 'project_id': '6d39b8d66e43454697d88a86755b45e3', 'start_time': datetime.datetime(2024, 5, 3, 20, 10, 49, 930003), 'finish_time': None, 'message': None}]
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi (Background on this error at: https://sqlalche.me/e/14/gkpj)
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: ERROR nova.api.openstack.wsgi
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: INFO nova.api.openstack.wsgi [None req-19b3c8fc-586e-40ad-b3b7-3cc604632d69 tempest-DeleteServersTestJSON-1007295085 tempest-DeleteServersTestJSON-1007295085-project-member] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: <class 'oslo_db.exception.DBReferenceError'>
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: DEBUG nova.api.openstack.wsgi [None req-19b3c8fc-586e-40ad-b3b7-3cc604632d69 tempest-DeleteServersTestJSON-1007295085 tempest-DeleteServersTestJSON-1007295085-project-member] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: <class 'oslo_db.exception.DBReferenceError'> {{(pid=56087) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:943}}
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: INFO nova.api.openstack.requestlog [None req-19b3c8fc-586e-40ad-b3b7-3cc604632d69 tempest-DeleteServersTestJSON-1007295085 tempest-DeleteServersTestJSON-1007295085-project-member] 10.209.37.96 "DELETE /compute/v2.1/servers/61f3c713-8ef1-4281-a3db-f199197efda3" status: 500 len: 208 microversion: 2.1 time: 1.209063
May 03 20:10:51.149668 np0037436303 <email address hidden>[56087]: [pid: 56087|app: 0|req: 914/1850] 10.209.37.96 () {66 vars in 1379 bytes} [Fri May 3 20:10:49 2024] DELETE /compute/v2.1/servers/61f3c713-8ef1-4281-a3db-f199197efda3 => generated 208 bytes in 1211 msecs (HTTP/1.1 500) 9 headers in 391 bytes (1 switches on core 0)
Seen once in:-
https://b04549659f81127bccd5-1b102124c6fd0b6d71635117190cffc0.ssl.cf1.rackcdn.com/917262/2/check/neutron-ovs-tempest-multinode-full/a21ea6a/testr_results.html
this is pretty notrivial to resolve.
the exception indicates that the instance does not exist in the cell db yet
that means we have not completed scheduling and there is no way to interrupt the spawn request at this point.
form an API perspective we could tighten the precondition to reject the delete with a 409 but that is generally not desirable for billing reasons.
i.e. an user should always be able to delete a server to not incur ongoing charges.
a simple solution to the trace would be to catch the exception and then
return with a 409 instead of a 500 allowing the client to retry.
this is different then modifying the preconditons and returning a 409 in all cases as we would only return a 409 during the period where a delete is not possible.
the other more complex approach would be to create an minimal instance record in the delete state.
or somehow modify the build_request record in the API db such that it will cause the spwan to fail later after schduing but before the call to the comptue node to actually boot the instance.
this is unfortunelty not a simple bug to fix properly.