openstack usage list throws API error

[bjolo]

env: kolla-ansible newton release

after archiving 10.000.000 deleted rows, openstack usage list throws error

steps done
> openstack usage list -> works and gives correct output
> nova-manage db archive_deleted_rows --max-rows 500000
command above executed multiple times, but not all deleted rows have been archived. There are more rows that should be archived
> openstack usage list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'> (HTTP 500) (Request-ID: req-e6ef0ddb-8377-4974-b342-61bbfc7dc70c)

from the nova-api log

2017-12-22 15:05:03.151 36 INFO nova.metadata.wsgi.server [-] 136.225.154.5,136.225.154.7 "GET /latest/meta-data/public-ipv4 HTTP/1.1" status: 200 len: 131 time: 0.0058219
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] Unexpected exception in API method
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions detailed=detailed)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions info['memory_mb'] = instance.flavor.memory_mb
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions self.obj_load_attr(name)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions self._load_flavor()
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions expected_attrs=['flavor', 'system_metadata'])
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions use_slave=use_slave)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions raise exception.InstanceNotFound(instance_id=uuid)
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions InstanceNotFound: Instance d9e7610b-d154-4902-9a72-7d4d3171586a could not be found.
2017-12-22 15:05:30.732 21 ERROR nova.api.openstack.extensions
2017-12-22 15:05:30.735 21 INFO nova.api.openstack.wsgi [req-a5e6d65c-53b1-4337-bf26-44b395741ea9 45d743d179794d3a83b18d8929c6b030 37a7881ba8544064829e5dd53b91e2b9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'>

[bjolo (bjorn-lofdahl)]

After running the archive operation repeatedly until there was nothing more to archive resolved the issue. No more InstanceNotFound errors

[Takashi Natsume (natsume-takashi)]

The bug is very similar to the following bug.

TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer
https://bugs.launchpad.net/nova/+bug/1643444

But it has been fixed in stable/newton.

Catch InstanceNotFound exception
https://review.openstack.org/#/c/467321/

Needs more information.
What is a commit ID?
Has the fix for bug 1643444 already been applied in your environment?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]