Tests: lazy load operation of attribute 'snapshot_metadata' cannot proceed

[Eric Harney]

Possible issue with the Snapshot object?

master @ 7f498b0 Merge "Obtain target authentication from database same as LIO target"

Running this tox env (subset of our full tests):

[testenv:py34not]
commands = ostestr {posargs} --concurrency=16 --regex (cinder.tests.unit.test_volume|cinder.tests.unit.image.test_cache)

cinder.tests.unit.test_volume.GetActiveByWindowTestCase.test_snapshot_get_active_by_window
------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "cinder/tests/unit/test_volume.py", line 5982, in test_snapshot_get_active_by_window
        datetime.datetime(1, 4, 1, 1, 1, 1)).objects
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 171, in wrapper
        result = fn(cls, context, *args, **kwargs)
      File "cinder/objects/snapshot.py", line 251, in get_active_by_window
        snapshots, expected_attrs=['metadata'])
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 991, in obj_make_list
        **extra_args)
      File "cinder/objects/snapshot.py", line 121, in _from_db_object
        metadata = db_snapshot.get('snapshot_metadata')
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 68, in get
        return getattr(self, key, default)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
        return self.impl.get(instance_state(instance), dict_)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 578, in get
        value = self.callable_(state, passive)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/strategies.py", line 502, in _load_for_state
        (orm_util.state_str(state), self.key)
    sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Snapshot at 0x7f38f90ae790> is not bound to a Session; lazy load operation of attribute 'snapshot_metadata' cannot proceed

[Eric Harney (eharney)]

Note that this was actually on python 2.7.

[Eric Harney (eharney)]

http://logs.openstack.org/10/230510/2/check/gate-cinder-python27/9f13d35/console.html

[Eric Harney (eharney)]

http://logs.openstack.org/47/229947/2/check/gate-cinder-python34/7ff6502/console.html

[Eric Harney (eharney)]

http://logs.openstack.org/22/231122/4/check/gate-cinder-python34/fae2045/console.html

[John Griffith (john-griffith)]

https://jenkins07.openstack.org/job/gate-cinder-python34/1421/console

[Eric Harney (eharney)]

http://logs.openstack.org/16/232216/10/check/gate-cinder-python34/952de4b/console.html

[Sean McGinnis (sean-mcginnis)]

I haven't seen this lately. Fixed indirectly by a dependency update?

[Michal Dulko (michal-dulko-f)]

I've hit it on my two patches just yesterday on both py27 and py34:

http://logs.openstack.org/87/238487/1/check/gate-cinder-python34/273654d/console.html
http://logs.openstack.org/18/238618/3/check/gate-cinder-python27/ca9897f/console.html

I've got that in backlog, so taking look at the issue now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238866

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238866
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ab6e2237bf464ec0c4c432ec6047a98cb30db6c5
Submitter: Jenkins
Branch: master

commit ab6e2237bf464ec0c4c432ec6047a98cb30db6c5
Author: Michał Dulko <email address hidden>
Date: Fri Oct 23 11:59:48 2015 +0200

    Eager load snapshot_metadata in *snapshot_get_all

    All methods returning snapshot lists in db.sqlalchemy.api are eager
    loading snapshot_metadata - besides snapshot_get_all_by_project and
    snapshot_get_active_by_window. In case of the latter that fact caused
    unit tests to randomly fail because of SQLAlchemy Session sometimes
    getting closed before the metadata got lazy loaded. This commit adds
    missing options(joinedload('snapshot_metadata')) to these queries.

    Change-Id: I72557ebc7af9f3044046965ca79c9fe7c15520fc
    Closes-Bug: 1501838

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/248035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/248035
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=586727cef755abeb3360f176782c1b1dd680d318
Submitter: Jenkins
Branch: master

commit 586727cef755abeb3360f176782c1b1dd680d318
Author: Michał Dulko <email address hidden>
Date: Fri Nov 20 12:36:22 2015 +0100

    Eager load columns in volume_get_active_by_window

    All other volume-related methods in db.sqlalchemy.api are eager loading
    volume_metadata, volume_admin_metadata (if admin), volume_type,
    volume_attachment and consistencygroup. volume_get_active_by_window
    wasn't, causing fails because of SQLAlchemy Session being closed when
    trying to lazy load volume_metadata. This commit adds missing
    options(joinedload()) to this query.

    Change-Id: I33ec89d7f1f43aae6010aaa6e12951b7c522b8f0
    Closes-Bug: 1517763
    Related-Bug: 1501838

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/cinder 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/304446

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304446
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7e970a1da65431655179b3bba084f4a63fdae959
Submitter: Jenkins
Branch: stable/liberty

commit 7e970a1da65431655179b3bba084f4a63fdae959
Author: Michał Dulko <email address hidden>
Date: Fri Nov 20 12:36:22 2015 +0100

    Eager load columns in volume_get_active_by_window

    All other volume-related methods in db.sqlalchemy.api are eager loading
    volume_metadata, volume_admin_metadata (if admin), volume_type,
    volume_attachment and consistencygroup. volume_get_active_by_window
    wasn't, causing fails because of SQLAlchemy Session being closed when
    trying to lazy load volume_metadata. This commit adds missing
    options(joinedload()) to this query.

    Change-Id: I33ec89d7f1f43aae6010aaa6e12951b7c522b8f0
    Closes-Bug: 1517763
    Related-Bug: 1501838
    (cherry picked from commit 586727cef755abeb3360f176782c1b1dd680d318)

[Tobias Urdin (tobias-urdin)]

One of them was backported into liberty, however this one still needs to be backported https://review.openstack.org/238866
We should merge the above review into stable liberty aswell, I'm running the review live in our liberty production since some months back

[Tobias Urdin (tobias-urdin)]

This was not merged for Liberty, adding Ubuntu release team to see if we can get such a patch merged into Ubuntu cloud archive if fixed since Liberty is now only security supported

[Corey Bryant (corey.bryant)]

Hi Tobias, Thanks for the notification on this. It looks like a reasonable backport to the liberty cinder package for ubuntu. We'll work on getting it backported. For now, marking this as fix-released for Xenial+ since both patches landed in 8.0.0.