race in server show api

[Christopher Yeoh]

Because of the instance object lazy loading its possible to get into situations where the API code is half way through assembling data to return to the client when the instance disappears underneath it. We really need to ensure everything we will need is retreived up front so we have a consistent snapshot view of the instance

[req-5ca39eb3-c1d2-433b-8dac-1bf5f338ce1f ServersAdminNegativeV3Test-1453501114 ServersAdminNegativeV3Test-364813115] Unexpected exception in API method
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 473, in wrapped
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions return f(*args, **kwargs)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/plugins/v3/servers.py", line 410, in show
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions return self._view_builder.show(req, instance)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 268, in show
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions _inst_fault = self._get_fault(request, instance)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/views/servers.py", line 214, in _get_fault
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions fault = instance.fault
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/base.py", line 67, in getter
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions self.obj_load_attr(name)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/instance.py", line 520, in obj_load_attr
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions expected_attrs=[attrname])
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/base.py", line 153, in wrapper
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/instance.py", line 310, in get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/api.py", line 676, in instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions columns_to_join, use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1715, in instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions columns_to_join=columns_to_join, use_slave=use_slave)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1727, in _instance_get_by_uuid
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions raise exception.InstanceNotFound(instance_id=uuid)
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions InstanceNotFound: Instance fcff276a-d410-4760-9b98-4014024b1353 could not be found.
2014-08-04 06:37:25.738 21228 TRACE nova.api.openstack.extensions

  http://logs.openstack.org/periodic-qa/periodic-tempest-dsvm-nova-v3-full-master/a278802/logs/screen-n-api.txt

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112192

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113947

[Matt Riedemann (mriedem)]

Yeah this shows up in the periodic-qa and experimental queues on non-negative API tests:

message:"Unexpected exception in API method" AND message:"show" AND message:"InstanceNotFound" AND tags:"screen-n-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVW5leHBlY3RlZCBleGNlcHRpb24gaW4gQVBJIG1ldGhvZFwiIEFORCBtZXNzYWdlOlwic2hvd1wiIEFORCBtZXNzYWdlOlwiSW5zdGFuY2VOb3RGb3VuZFwiIEFORCB0YWdzOlwic2NyZWVuLW4tYXBpLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA4MTMyNjM2ODkxLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113947
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=257183a8a9130f2b444f7f96ec8582da89684528
Submitter: Jenkins
Branch: master

commit 257183a8a9130f2b444f7f96ec8582da89684528
Author: Dan Smith <email address hidden>
Date: Wed Aug 13 09:03:49 2014 -0700

    Direct-load Instance.fault when lazy-loading

    This breaks out instance.fault lazy-loading from the other attributes,
    since we have a direct and more efficient way of fetching the fault than
    re-querying the entire instance with the fault attached.

    This also should help address fault-related races in the API where a
    list of instances is queried, one of those is deleted, and the fault
    attribute later triggers an InstanceNotFound whilst trying to do the
    lazy-load.

    Change-Id: Iceb552663db93fa2a01fb90ece0c1eebecdb783f
    Closes-bug: #1352659

[OpenStack Infra (hudson-openstack)]

Change abandoned by Christopher Yeoh (<email address hidden>) on branch: master
Review: https://review.openstack.org/112192
Reason: Addressed by https://review.openstack.org/#/c/113947/ instead