TenantUsagesTestJSON.test_list_usage_all_tenants 500 from Db layer

[Attila Fazekas]

I have a newton setup with 3 api(controller) node.

TenantUsagesTestJSON.test_list_usage_all_tenants failed once , the failure looks similar to one described in the already fixed in https://bugs.launchpad.net/nova/+bug/1487570 , but it is different api call so it can have similar issue.

Likely you have an old list of ids, and trying to fetch more info about an already deleted instance.

The tempest exception:

2016-11-20 00:07:18,606 27600 INFO [tempest.lib.common.rest_client] Request (TenantUsagesTestJSON:test_list_usage_all_tenants): 500 GET http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 0.134s
2016-11-20 00:07:18,607 27600 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '500', 'content-length': '205', 'content-location': 'http://[2620:52:0:13b8:5054:ff:fe3e:4]:8774/v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313', 'x-compute-request-id': 'req-3ff84c48-b03e-4f23-8f33-227719a0ced4', 'vary': 'X-OpenStack-Nova-API-Version', 'openstack-api-version': 'compute 2.1', 'connection': 'close', 'x-openstack-nova-api-version': '2.1', 'date': 'Sun, 20 Nov 2016 05:07:18 GMT', 'content-type': 'application/json; charset=UTF-8'}
        Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.InstanceNotFound'>", "code": 500}}
  File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 73, in test_list_usage_all_tenants
    start=self.start, end=self.end, detailed="1")['tenant_usages'][0]
  File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 63, in call_until_valid
    self.assertEqual(test_utils.call_until_true(is_valid, duration, 1),
  File "/home/stack/tempest-dir/tempest/lib/common/utils/test_utils.py", line 103, in call_until_true
    if func():
  File "/home/stack/tempest-dir/tempest/api/compute/admin/test_simple_tenant_usage.py", line 59, in is_valid
    self.resp = func(*args, **kwargs)
  File "/home/stack/tempest-dir/tempest/lib/services/compute/tenant_usages_client.py", line 37, in list_tenant_usages
    resp, body = self.get(url)
  File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 291, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/stack/tempest-dir/tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 664, in request
    self._error_checker(resp, resp_body)
  File "/home/stack/tempest-dir/tempest/lib/common/rest_client.py", line 827, in _error_checker
    message=message)
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'>

The related nova api log (node-2):
2016-11-20 05:07:18.476 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Calling method '<bound method SimpleTenantUsageController.index of <nova.api.openstack.compute.simple_tenant_usage.SimpleTenantUsageController object at 0x7fc62d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2016-11-20 05:07:18.507 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013
2016-11-20 05:07:18.556 111884 DEBUG nova.objects.instance [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Lazy-loading 'flavor' on Instance uuid 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1013
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Unexpected exception in API method
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 238, in index
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions detailed=detailed)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/simple_tenant_usage.py", line 123, in _tenant_usages_for_period
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions info['memory_mb'] = instance.flavor.memory_mb
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions self.obj_load_attr(name)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1041, in obj_load_attr
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions self._load_flavor()
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 863, in _load_flavor
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions expected_attrs=['flavor', 'system_metadata'])
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 457, in get_by_uuid
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions use_slave=use_slave)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 449, in _db_instance_get_by_uuid
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 725, in instance_get_by_uuid
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1876, in _instance_get_by_uuid
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions raise exception.InstanceNotFound(instance_id=uuid)
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions InstanceNotFound: Instance 5f3a04c2-ab22-4378-9512-bfd4f9fb0a52 could not be found.
2016-11-20 05:07:18.597 111884 ERROR nova.api.openstack.extensions
2016-11-20 05:07:18.601 111884 INFO nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'>
2016-11-20 05:07:18.601 111884 DEBUG nova.api.openstack.wsgi [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceNotFound'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044
2016-11-20 05:07:18.602 111884 INFO nova.osapi_compute.wsgi.server [req-3ff84c48-b03e-4f23-8f33-227719a0ced4 d4852c5eaf2645e2aab0c7485939395a 998c56750d4a4056853829f088ce2be9 - default default] fd00:fd00:fd00:2000::11 "GET /v2.1/os-simple-tenant-usage?detailed=1&start=2016-11-19T00%3A07%3A17.645313&end=2016-11-21T00%3A07%3A17.645313 HTTP/1.1" status: 500 len: 567 time: 0.1304212
2016-11-20 05:07:18.929 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?tenant_id=72ac28353e7f4c03a79ba6abd32c189a&shared=False used request id req-9f2b5b0f-5b12-4cb4-981a-000e3bb3fe8c _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127
2016-11-20 05:07:18.989 111891 DEBUG neutronclient.v2_0.client [req-0063c383-52da-4238-bda6-96079822b4a9 e87a6c64041247c186f36fbeaa1987cb 72ac28353e7f4c03a79ba6abd32c189a - default default] GET call to neutron for http://[fd00:fd00:fd00:2000::18]:9696/v2.0/networks.json?shared=True used request id req-4bdabb6e-cbd8-449d-a4b8-61355625bd88 _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127

[Attila Fazekas (afazekas)]

nova-api.log.2 is the log from the related node, which has the related exception.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/400399

[OpenStack Infra (hudson-openstack)]

Change abandoned by jichenjc (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/400399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/400763

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/400763
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=83e6dbf5815e2b0920e3afa404efbc46c09077e9
Submitter: Jenkins
Branch: master

commit 83e6dbf5815e2b0920e3afa404efbc46c09077e9
Author: jichenjc <email address hidden>
Date: Fri Nov 18 18:02:11 2016 +0800

    Catch InstanceNotFound exception

    during simple_tenant_usage collection, there is possibility that
    instance got delete, so we need to catch the exception because
    instance.flavor might cause a instance.get_by_uuid call.

    commit b52876f912156be3f55d1c78898f9050ec498b5a mentioned
    <This follows the previous patch and makes simple-tenant-usage use
    the bundled flavor for basic instance resource usage like everything
    else. This is an important step to being able to remove those fields
    at some point>
    so this patch didn't use the flavor object directly

    Change-Id: Icbfd1a4b97873d44bafd16754d09fff7a9d2b7af
    Closes-Bug: 1643444

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/467321

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/468319

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/468514

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/482219

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artom Lifshitz (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/482219

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/499208

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/468514
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7ab19c9e78bf1c4de58f85092e016c93666b90e7
Submitter: Jenkins
Branch: master

commit 7ab19c9e78bf1c4de58f85092e016c93666b90e7
Author: Artom Lifshitz <email address hidden>
Date: Fri May 26 14:28:32 2017 -0400

    Test InstanceNotFound handling in 'nova usage'

    83e6dbf5815e2b0920e3afa404efbc46c09077e9 added handling of
    InstanceNotFound in simple tenant usage, but did not contain tests.
    This patch adds a test.

    Related-bug: 1643444
    Related-bug: 1692893
    Change-Id: Ia5c407d9038e2da84a70d7cc5be531ea7ef714f1

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/468319
Reason: Urgh, so the change-id here is for another change that I think we originally tried to squash together with the unit tests? As we don't need to do that for Ocata I'm going to abandon this change and have Artom reopen https://review.openstack.org/#/c/482219/ .

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499208
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c981a27cd321306e00a3d48945d1beb10346b8df
Submitter: Jenkins
Branch: stable/pike

commit c981a27cd321306e00a3d48945d1beb10346b8df
Author: Artom Lifshitz <email address hidden>
Date: Fri May 26 14:28:32 2017 -0400

    Test InstanceNotFound handling in 'nova usage'

    83e6dbf5815e2b0920e3afa404efbc46c09077e9 added handling of
    InstanceNotFound in simple tenant usage, but did not contain tests.
    This patch adds a test.

    Related-bug: 1643444
    Related-bug: 1692893
    Change-Id: Ia5c407d9038e2da84a70d7cc5be531ea7ef714f1
    (cherry picked from commit 7ab19c9e78bf1c4de58f85092e016c93666b90e7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/482219
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=14eeaf1c5203ea2b843a44a7130b76938a396e8e
Submitter: Zuul
Branch: stable/ocata

commit 14eeaf1c5203ea2b843a44a7130b76938a396e8e
Author: Artom Lifshitz <email address hidden>
Date: Fri May 26 14:28:32 2017 -0400

    Test InstanceNotFound handling in 'nova usage'

    83e6dbf5815e2b0920e3afa404efbc46c09077e9 added handling of
    InstanceNotFound in simple tenant usage, but did not contain tests.
    This patch adds a test.

    Related-bug: 1643444
    Related-bug: 1692893
    Change-Id: Ia5c407d9038e2da84a70d7cc5be531ea7ef714f1
    (cherry picked from commit 7ab19c9e78bf1c4de58f85092e016c93666b90e7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467321
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=06584dc10f59fc7d4fc953fa93f7b4bb6ffc5123
Submitter: Zuul
Branch: stable/newton

commit 06584dc10f59fc7d4fc953fa93f7b4bb6ffc5123
Author: jichenjc <email address hidden>
Date: Fri Nov 18 18:02:11 2016 +0800

    Catch InstanceNotFound exception

    during simple_tenant_usage collection, there is possibility that
    instance got delete, so we need to catch the exception because
    instance.flavor might cause a instance.get_by_uuid call.

    commit b52876f912156be3f55d1c78898f9050ec498b5a mentioned
    <This follows the previous patch and makes simple-tenant-usage use
    the bundled flavor for basic instance resource usage like everything
    else. This is an important step to being able to remove those fields
    at some point>
    so this patch didn't use the flavor object directly

    Backport note: since the original fix did not include tests, this
    backport squashes 14eeaf1c5203ea2b843a44a7130b76938a396e8e into
    itself.

    Change-Id: Icbfd1a4b97873d44bafd16754d09fff7a9d2b7af
    Closes-Bug: 1643444
    Closes-Bug: 1692893
    (cherry picked from commit 83e6dbf5815e2b0920e3afa404efbc46c09077e9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.9 release.