detached instance error on some unit tests

[Sean Dague]

Caught in an unrelated change set in the unit tests - http://logs.openstack.org/18147/4/check/gate-nova-python26/12438/testr_results.html

It looks like there is another lazy loaded dependent object which is now failing due to the no-db-compute changes.

[Dan Smith (danms)]

Notes so far:

Actually, it's failing on the "deleted" column, which is neither lazy-loaded nor specific to Instance.

Also, I can reproduce it (seemingly) all the time on my fast machine, and (seemingly) never on a slow one, and in both cases, for full runs and single tests.

It's failing to convert the instance on the outbound side of the RPC call from nova-api, which means that the code making the call is passing some half-baked instance or something, and isn't related to no-db-compute changes, but rather no-db-messaging ones which have been in the tree for quite a while.

I tend to think this is either some race that just got uncovered, or the result of something else changing...

[Dan Smith (danms)]

Correction, it's the deleted property of the Instance.metadata (InstanceMetadata) object, which _is_ lazy-loaded from Instance. It looks to me like the problem arises in the instance that is returned from self.update() during rebuild, which is db.instance_update_and_get_original().

[Dan Smith (danms)]

The following tweak fixes it for me, and I think it's related to the same issue as noted by comstud above, where sqlalchemy fails to refresh the instance['metadata'] after the update. However, I don't understand why the same session.refresh() doesn't work for me, so that will have to wait for tomorrow and someone who understands it more than me.

diff --git a/nova/db/sqlalchemy/api.py b/nova/db/sqlalchemy/api.py
index dce92ba..a0a73e8 100644
--- a/nova/db/sqlalchemy/api.py
+++ b/nova/db/sqlalchemy/api.py
@@ -1897,6 +1897,13 @@ def _instance_update(context, instance_uuid, values, copy
             # instance_type.
             session.refresh(instance_ref['instance_type'])

+ if metadata is not None:
+ # Why can't I do this:
+ # session.refresh(instance_ref['metadata'])
+ # ?
+ instance_ref['metadata'] = _instance_metadata_get_query(
+ context, instance_ref['uuid'], session=session).all()
+
     return (old_instance_ref, instance_ref)

[Chris Behrens (cbehrens)]

Good question on why refresh doesn't work. sqlalchemy may think everything is up to date, perhaps.

I don't really understand why InstanceMetadata is trying to be lazy-loaded... unless sqlalchemy thinks its out of date for some reason. But if the above patch fixes it, I'm guessing the below patch will as well. I like this below patch better as it doesn't add any queries. Give this a try?

diff --git a/nova/db/sqlalchemy/api.py b/nova/db/sqlalchemy/api.py
index dce92ba..de5003d 100644
--- a/nova/db/sqlalchemy/api.py
+++ b/nova/db/sqlalchemy/api.py
@@ -1877,13 +1877,13 @@ def _instance_update(context, instance_uuid, values, cop

         metadata = values.get('metadata')
         if metadata is not None:
- instance_metadata_update(context, instance_ref['uuid'],
- values.pop('metadata'), True,
- session=session)
+ md_models = instance_metadata_update(context,
+ instance_ref['uuid'], values.pop('metadata'), True,
+ session=session)

         system_metadata = values.get('system_metadata')
         if system_metadata is not None:
- instance_system_metadata_update(
+ sys_md_models = instance_system_metadata_update(
                  context, instance_ref['uuid'], values.pop('system_metadata'),
                  delete=True, session=session)

@@ -1896,6 +1896,10 @@ def _instance_update(context, instance_uuid, values, copy
             # but the rest of the model has the data from the old
             # instance_type.
             session.refresh(instance_ref['instance_type'])
+ if metadata is not None:
+ instance_ref['metadata'] = md_models
+ if system_metadata is not None:
+ instance_ref['system_metadata'] = sys_md_models

     return (old_instance_ref, instance_ref)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/19135

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19135
Committed: http://github.com/openstack/nova/commit/321a8396ca6e0a6896658e9ceb724f1ba796ee40
Submitter: Jenkins
Branch: master

commit 321a8396ca6e0a6896658e9ceb724f1ba796ee40
Author: Dan Smith <email address hidden>
Date: Mon Jan 7 14:17:07 2013 -0500

    Refresh instance metadata in-place

    For some reason that isn't entirely clear to me, the attempt to update
    instance metadata in _instance_update() results in a stale sqlalchemy
    object state. This is a similar issue to the one fixed in commit:

      9fdf7552779d518af9cda4e366bf81fddb0cb6f2

    For some reason, session.refresh() doesn't work for instance['metadata']
    in the same way, which also raises suspicion that perhaps it's not the
    right fix for system_metadata either.

    This patch adds an _instance_update_metadata_in_place() method that
    updates the actual instance['metadata'] list _and_ the database,
    mirroring the high-level behavior of instance_metadata_update(), but
    without requiring us to refresh or re-fetch the metadata or the
    whole instance (as a workaround).

    Fixes bug 1096653

    Change-Id: Ic5a205631b1b7dce3744960ed4201dcc7b4b2ae6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19263
Committed: http://github.com/openstack/nova/commit/35328ddd8fd1aadf0800ae282a7654c106bc52f3
Submitter: Jenkins
Branch: master

commit 35328ddd8fd1aadf0800ae282a7654c106bc52f3
Author: Giampaolo Lauria <email address hidden>
Date: Tue Jan 8 17:43:51 2013 -0500

    Added sample tests to FlavorSwap API.

    Partially implements blueprint nova-api-samples
    Fixes bug 1096653

    Change-Id: I9a6ca8a023bf0f3404b3a9af635dd29a7a584aa3