nova list --deleted fails with a 500 response

[Matt Riedemann]

Nova version (mitaka):

stack@archive:/opt/stack/nova$ git log -1
commit f6a5a43e06c2af6325d7a3552c71e968565684fc
Merge: f268cf5 0df6fba
Author: Jenkins <email address hidden>
Date: Mon Nov 16 18:01:40 2015 +0000

    Merge "Remove duplicate server.kill on test shutdown"
stack@archive:/opt/stack/nova$

python-novaclient version:

stack@archive:/opt/stack/nova$ pip show python-novaclient
---
Metadata-Version: 2.0
Name: python-novaclient
Version: 2.35.0
Summary: Client library for OpenStack Compute API
Home-page: https://www.openstack.org
Author: OpenStack
Author-email: <email address hidden>
License: Apache License, Version 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: oslo.i18n, oslo.serialization, python-keystoneclient, argparse, Babel, oslo.utils, iso8601, requests, pbr, six, PrettyTable, simplejson
stack@archive:/opt/stack/nova$

As a non-admin user, I created some servers and deleted them:

mysql> select display_name,uuid,deleted from nova.instances;
+--------------+--------------------------------------+---------+
| display_name | uuid | deleted |
+--------------+--------------------------------------+---------+
| test | ca3e57c0-cf37-40ab-9322-4cde6fe242d9 | 1 |
| test2 | 8d555e3b-41db-495d-8d03-4918a011472c | 2 |
| test3 | ef4c5911-958a-47e7-b836-42da9d32c209 | 3 |
+--------------+--------------------------------------+---------+
3 rows in set (0.00 sec)

As the admin user, I should be able to list them using 'nova list --deleted', but that fails with a 500 because an InstanceNotFound is not handled in the API code:

2015-11-20 16:26:11.033 ERROR nova.api.openstack.extensions [req-aaea6f78-bf6d-4c35-ad46-decba714767b admin demo] Unexpected exception in API method
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 280, in detail
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 406, in _get_servers
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions response = self._view_builder.detail(req, instance_list)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 151, in detail
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions return self._list_view(self.show, request, instances, coll_name)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 163, in _list_view
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions server_list = [func(request, server)["server"] for server in servers]
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 291, in show
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions "flavor": self._get_flavor(request, instance),
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 223, in _get_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions instance_type = instance.get_flavor()
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 863, in get_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions return getattr(self, attr)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 66, in getter
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions self.obj_load_attr(name)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 853, in obj_load_attr
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions self._load_flavor()
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 744, in _load_flavor
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions expected_attrs=['flavor', 'system_metadata'])
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 171, in wrapper
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 373, in get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 651, in instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions columns_to_join, use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1719, in instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions columns_to_join=columns_to_join, use_slave=use_slave)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1731, in _instance_get_by_uuid
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions raise exception.InstanceNotFound(instance_id=uuid)
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions InstanceNotFound: Instance ef4c5911-958a-47e7-b836-42da9d32c209 could not be found.
2015-11-20 16:26:11.033 TRACE nova.api.openstack.extensions

It looks like it's trying to lazy load a flavor on the instance object which fails because it's not reading the deleted instance at that point.

[Matt Riedemann (mriedem)]

I should note that before running `nova list --deleted` here I was also doing some archiving, so the DB might be in a weird state if there are half-missing dependent resources for loading the instance to work, i.e. running:

nova-manage db archive_deleted_rows 5000

[Matt Riedemann (mriedem)]

This is invalid, it's a side effect of bug 1183523 where archive_deleted_rows deletes some things but not others (like instances) because of foreign key constraints.

So the problem I was having was the instance_extra table was empty, and that's where the flavor information is stored for the instance.

When I cleaned my DB and re-did the scenario:

1. boot instance
2. delete instance
3. nova list --deleted

That works (as admin for #3). So marking this invalid.