Tests: lazy load operation of attribute 'snapshot_metadata' cannot proceed

Bug #1501838 reported by Eric Harney on 2015-10-01
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Critical
Michal Dulko
cinder (Ubuntu)
High
Unassigned
Wily
High
Unassigned
Xenial
High
Unassigned

Bug Description

Possible issue with the Snapshot object?

master @ 7f498b0 Merge "Obtain target authentication from database same as LIO target"

Running this tox env (subset of our full tests):

[testenv:py34not]
commands = ostestr {posargs} --concurrency=16 --regex (cinder.tests.unit.test_volume|cinder.tests.unit.image.test_cache)

cinder.tests.unit.test_volume.GetActiveByWindowTestCase.test_snapshot_get_active_by_window
------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "cinder/tests/unit/test_volume.py", line 5982, in test_snapshot_get_active_by_window
        datetime.datetime(1, 4, 1, 1, 1, 1)).objects
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 171, in wrapper
        result = fn(cls, context, *args, **kwargs)
      File "cinder/objects/snapshot.py", line 251, in get_active_by_window
        snapshots, expected_attrs=['metadata'])
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 991, in obj_make_list
        **extra_args)
      File "cinder/objects/snapshot.py", line 121, in _from_db_object
        metadata = db_snapshot.get('snapshot_metadata')
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 68, in get
        return getattr(self, key, default)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
        return self.impl.get(instance_state(instance), dict_)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 578, in get
        value = self.callable_(state, passive)
      File "/home/eharney/src/cinder-trunk8/.tox/py34not/lib/python2.7/site-packages/sqlalchemy/orm/strategies.py", line 502, in _load_for_state
        (orm_util.state_str(state), self.key)
    sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Snapshot at 0x7f38f90ae790> is not bound to a Session; lazy load operation of attribute 'snapshot_metadata' cannot proceed

Eric Harney (eharney) wrote :
Eric Harney (eharney) wrote :

Note that this was actually on python 2.7.

John Griffith (john-griffith) wrote :
Changed in cinder:
importance: Undecided → Critical
status: New → Confirmed
Sean McGinnis (sean-mcginnis) wrote :

I haven't seen this lately. Fixed indirectly by a dependency update?

Michal Dulko (michal-dulko-f) wrote :

I've hit it on my two patches just yesterday on both py27 and py34:

http://logs.openstack.org/87/238487/1/check/gate-cinder-python34/273654d/console.html
http://logs.openstack.org/18/238618/3/check/gate-cinder-python27/ca9897f/console.html

I've got that in backlog, so taking look at the issue now.

Fix proposed to branch: master
Review: https://review.openstack.org/238866

Changed in cinder:
assignee: nobody → Michal Dulko (michal-dulko-f)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/238866
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ab6e2237bf464ec0c4c432ec6047a98cb30db6c5
Submitter: Jenkins
Branch: master

commit ab6e2237bf464ec0c4c432ec6047a98cb30db6c5
Author: Michał Dulko <email address hidden>
Date: Fri Oct 23 11:59:48 2015 +0200

    Eager load snapshot_metadata in *snapshot_get_all

    All methods returning snapshot lists in db.sqlalchemy.api are eager
    loading snapshot_metadata - besides snapshot_get_all_by_project and
    snapshot_get_active_by_window. In case of the latter that fact caused
    unit tests to randomly fail because of SQLAlchemy Session sometimes
    getting closed before the metadata got lazy loaded. This commit adds
    missing options(joinedload('snapshot_metadata')) to these queries.

    Change-Id: I72557ebc7af9f3044046965ca79c9fe7c15520fc
    Closes-Bug: 1501838

Changed in cinder:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/248035
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=586727cef755abeb3360f176782c1b1dd680d318
Submitter: Jenkins
Branch: master

commit 586727cef755abeb3360f176782c1b1dd680d318
Author: Michał Dulko <email address hidden>
Date: Fri Nov 20 12:36:22 2015 +0100

    Eager load columns in volume_get_active_by_window

    All other volume-related methods in db.sqlalchemy.api are eager loading
    volume_metadata, volume_admin_metadata (if admin), volume_type,
    volume_attachment and consistencygroup. volume_get_active_by_window
    wasn't, causing fails because of SQLAlchemy Session being closed when
    trying to lazy load volume_metadata. This commit adds missing
    options(joinedload()) to this query.

    Change-Id: I33ec89d7f1f43aae6010aaa6e12951b7c522b8f0
    Closes-Bug: 1517763
    Related-Bug: 1501838

This issue was fixed in the openstack/cinder 8.0.0.0b1 development milestone.

Thierry Carrez (ttx) on 2015-12-03
Changed in cinder:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/304446
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7e970a1da65431655179b3bba084f4a63fdae959
Submitter: Jenkins
Branch: stable/liberty

commit 7e970a1da65431655179b3bba084f4a63fdae959
Author: Michał Dulko <email address hidden>
Date: Fri Nov 20 12:36:22 2015 +0100

    Eager load columns in volume_get_active_by_window

    All other volume-related methods in db.sqlalchemy.api are eager loading
    volume_metadata, volume_admin_metadata (if admin), volume_type,
    volume_attachment and consistencygroup. volume_get_active_by_window
    wasn't, causing fails because of SQLAlchemy Session being closed when
    trying to lazy load volume_metadata. This commit adds missing
    options(joinedload()) to this query.

    Change-Id: I33ec89d7f1f43aae6010aaa6e12951b7c522b8f0
    Closes-Bug: 1517763
    Related-Bug: 1501838
    (cherry picked from commit 586727cef755abeb3360f176782c1b1dd680d318)

tags: added: in-stable-liberty
Tobias Urdin (tobias-urdin) wrote :

One of them was backported into liberty, however this one still needs to be backported https://review.openstack.org/238866
We should merge the above review into stable liberty aswell, I'm running the review live in our liberty production since some months back

Tobias Urdin (tobias-urdin) wrote :

This was not merged for Liberty, adding Ubuntu release team to see if we can get such a patch merged into Ubuntu cloud archive if fixed since Liberty is now only security supported

Changed in cinder (Ubuntu):
status: New → Confirmed
Corey Bryant (corey.bryant) wrote :

Hi Tobias, Thanks for the notification on this. It looks like a reasonable backport to the liberty cinder package for ubuntu. We'll work on getting it backported. For now, marking this as fix-released for Xenial+ since both patches landed in 8.0.0.

Changed in cinder (Ubuntu Wily):
status: New → Triaged
Changed in cinder (Ubuntu Xenial):
status: New → Fix Released
Changed in cinder (Ubuntu):
status: Confirmed → Fix Released
importance: Undecided → High
Changed in cinder (Ubuntu Xenial):
importance: Undecided → High
Changed in cinder (Ubuntu):
importance: High → Undecided
Changed in cinder (Ubuntu Wily):
importance: Undecided → High
Changed in cinder (Ubuntu):
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments