DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session

Bug #1834845 reported by Patrick Oberdorf on 2019-07-01
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Gorka Eguileor
Ubuntu Cloud Archive
High
James Page
Stein
High
James Page
Train
High
James Page
cinder (Ubuntu)
High
James Page
Disco
High
James Page
Eoan
High
James Page

Bug Description

Hey there.

We upgraded from rocky to stein and facing a strange error. Our cinder-ceph volume services gets marked as down after a few seconds. So we looked into this and found a error in the logs:

Error starting thread.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service Traceback (most recent call last):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service.start()
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 222, in start
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service_id=Service.service_id)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 445, in init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 523, in _init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes = self._get_my_volumes(ctxt)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3007, in _get_my_volumes
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service limit, offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3003, in _get_my_resources
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service offset=offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 617, in get_all
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes, expected_attrs=expected_attrs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 290, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service db_volume.get('volume_attachment'))
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume_attachment.py", line 102, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service if 'volume' in expected_attrs and hasattr(db_attachment, 'volume'):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service return self.impl.get(instance_state(instance), dict_)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service value = self.callable_(state, passive)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/strategies.py", line 596, in _load_for_state
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service (orm_util.state_str(state), self.key)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

It looks like, because of this error the service gets marked down. This issue is definitly linked to this: https://bugs.launchpad.net/cinder/+bug/1812913 but the fix that is released is not fixing it.
If we put a try-catch block around this block, the services stays up, but i am not sure if something else will break.
This is in our perspective a high priority bug, because we are not able to use one of our ceph backends.

ii cinder-api 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-common 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:14.0.0-0ubuntu1~cloud0 all Cinder Python 3 libraries

Sean McGinnis (sean-mcginnis) wrote :

Gorka, could you take a look at this?

Changed in cinder:
assignee: nobody → Gorka Eguileor (gorka)
Patrick Oberdorf (obi12341) wrote :

_Dirty_ workaround for this

The attachment "really_dirty_workaround.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch

Fix proposed to branch: master
Review: https://review.opendev.org/668646

Changed in cinder:
status: New → In Progress
Patrick Oberdorf (obi12341) wrote :

I can confirm the commit is working :)

Changed in cinder:
assignee: Gorka Eguileor (gorka) → Eric Harney (eharney)
Eric Harney (eharney) on 2019-07-02
Changed in cinder:
assignee: Eric Harney (eharney) → Gorka Eguileor (gorka)
Changed in cinder:
assignee: Gorka Eguileor (gorka) → Eric Harney (eharney)
Eric Harney (eharney) on 2019-07-03
Changed in cinder:
assignee: Eric Harney (eharney) → Gorka Eguileor (gorka)
Changed in cinder:
assignee: Gorka Eguileor (gorka) → Eric Harney (eharney)
Eric Harney (eharney) on 2019-07-09
Changed in cinder:
assignee: Eric Harney (eharney) → Gorka Eguileor (gorka)
Matt Riedemann (mriedem) wrote :

Note that bug 1626499 and bug 1640920 seem like older versions of a similar thing but on different parts of the data model.

Reviewed: https://review.opendev.org/668646
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7
Submitter: Zuul
Branch: master

commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

    Fix DetachedInstanceError for VolumeAttachment

    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".

    This patch replaces "hasattr" with a check on the object's dictionary,
    which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867

Changed in cinder:
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cinder (Ubuntu):
status: New → Confirmed
tags: added: canonical-bootstack
James Page (james-page) on 2019-08-07
Changed in cinder (Ubuntu Eoan):
status: Confirmed → Triaged
Changed in cinder (Ubuntu Disco):
status: New → Triaged
importance: Undecided → High
Changed in cinder (Ubuntu Eoan):
importance: Undecided → High
James Page (james-page) on 2019-08-07
Changed in cinder (Ubuntu Eoan):
status: Triaged → In Progress
assignee: nobody → James Page (james-page)
James Page (james-page) wrote :

Ubuntu SRU information

[Impact]
After upgrading from Ubuntu with OpenStack Rocky to Ubuntu with OpenStack Stein, the cinder-volume service fails to restart resulting in loss of ability to create and manage volumes

[Test Case]
Deploy OpenStack Rocky
Create volume, attach to instance
Upgrade to OpenStack Stein
openstack volume create --size 5 test-volume fails.

[Regression Potential]
The patch for this fix is fairly defensive in approach and fixes an issue with an associated failure fixed in an earlier commit; low regression potential.

James Page (james-page) wrote :

I've uploaded fixes for eoan and disco; disco-proposed current contains 14.0.1-0ubuntu1 which will need to clear first.

Changed in cinder (Ubuntu Disco):
status: Triaged → In Progress
assignee: nobody → James Page (james-page)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1

---------------
cinder (2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1) eoan; urgency=medium

  * New upstream snapshot for OpenStack Train (LP: #1834845).

 -- James Page <email address hidden> Wed, 07 Aug 2019 13:13:56 +0100

Changed in cinder (Ubuntu Eoan):
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/675081
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Submitter: Zuul
Branch: stable/stein

commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

    Fix DetachedInstanceError for VolumeAttachment

    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".

    This patch replaces "hasattr" with a check on the object's dictionary,
    which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)

Corey Bryant (corey.bryant) wrote :

I've uploaded 2:14.0.1-0ubuntu3 to the disco unapproved queue which includes 2:14.0.1-0ubuntu2 and provides a fix for disco autopkgtest failure.

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Reviewed: https://review.opendev.org/680000
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=eabf648b7acce33cb93d667e8f6e70505093b4c9
Submitter: Zuul
Branch: stable/rocky

commit eabf648b7acce33cb93d667e8f6e70505093b4c9
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

    Fix DetachedInstanceError for VolumeAttachment

    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".

    This patch replaces "hasattr" with a check on the object's dictionary,
    which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)
    (cherry picked from commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b)
    Conflicts:
        cinder/tests/unit/volume/drivers/test_lvm_driver.py

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/685978
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2abd8a68bb1bde9adc7c870a1b827731ad4e42e9
Submitter: Zuul
Branch: stable/queens

commit 2abd8a68bb1bde9adc7c870a1b827731ad4e42e9
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

    Fix DetachedInstanceError for VolumeAttachment

    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".

    This patch replaces "hasattr" with a check on the object's dictionary,
    which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)
    (cherry picked from commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b)
    Conflicts:
        cinder/tests/unit/volume/drivers/test_lvm_driver.py
    (cherry picked from commit eabf648b7acce33cb93d667e8f6e70505093b4c9)

tags: added: in-stable-queens

This issue was fixed in the openstack/cinder 14.0.2 release.

This issue was fixed in the openstack/cinder 12.0.10 release.

Hello Patrick, or anyone else affected,

Accepted cinder into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed

This issue was fixed in the openstack/cinder 13.0.8 release.

Corey Bryant (corey.bryant) wrote :

This was fixed in Ubuntu Train version 2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1.

Corey Bryant (corey.bryant) wrote :

This is fixed in Ubuntu Disco version 2:14.0.1-0ubuntu2, so this is currently in disco-proposed.

Changed in cinder (Ubuntu Disco):
status: In Progress → Fix Committed
Corey Bryant (corey.bryant) wrote :

Since disco is EOL this month, we can work this through the stein cloud archive.

Changed in cinder (Ubuntu Disco):
status: Fix Committed → Won't Fix
Changed in cloud-archive:
status: Fix Committed → Fix Released
Corey Bryant (corey.bryant) wrote :

Verified successfully:

$ juju ssh cinder/0 apt policy cinder-common
cinder-common:
  Installed: 2:13.0.8-0ubuntu1~cloud0
  Candidate: 2:13.0.8-0ubuntu1~cloud0
  Version table:
 *** 2:13.0.8-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64

$ openstack volume list
+--------------------------------------+------+--------+------+-----------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+--------+------+-----------------------------+
| 760b2fe5-bfa3-4ec6-964d-6fd05ccba3b0 | v1 | in-use | 1 | Attached to x1 on /dev/vdb |
+--------------------------------------+------+--------+------+-----------------------------+

$ juju config cinder openstack-origin=cloud:bionic-stein/proposed

$ juju ssh cinder/0 apt policy cinder-common
cinder-common:
  Installed: 2:14.0.2-0ubuntu1~cloud0
  Candidate: 2:14.0.2-0ubuntu1~cloud0
  Version table:
 *** 2:14.0.2-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64

$ openstack volume create --size 5 v2

$ openstack volume list
+--------------------------------------+------+-----------+------+-----------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+-----------+------+-----------------------------+
| 41b29ac2-c580-444c-ba65-17d016fb136f | v2 | available | 5 | |
| 760b2fe5-bfa3-4ec6-964d-6fd05ccba3b0 | v1 | in-use | 1 | Attached to x1 on /dev/vdb |
+--------------------------------------+------+-----------+------+-----------------------------+

tags: added: verification-stein-done
removed: verification-stein-needed

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package cinder - 2:14.0.2-0ubuntu1~cloud0
---------------

 cinder (2:14.0.2-0ubuntu1~cloud0) bionic-stein; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 cinder (2:14.0.2-0ubuntu1) disco; urgency=medium
 .
   * New upstream point release for OpenStack Stein (LP: #1849192):
     - Includes fix for SIGABRT in cinder-backup/ceph (LP: #1838691).
     - d/p/Fix-DetachedInstanceError-for-VolumeAttachment.patch: Drop,
       included in release.
 .
 cinder (2:14.0.1-0ubuntu3) disco; urgency=medium
 .
   * d/tests/cinder-daemons: As of disco, sqlalchemy defaults to mysqldb so
     we must override the driver to use pymysql (LP: #1845321).
 .
 cinder (2:14.0.1-0ubuntu2) disco; urgency=medium
 .
   * d/p/Fix-DetachedInstanceError-for-VolumeAttachment.patch: Cherry
     pick fix for issue with DetachedInstanceError being generated due
     to compatibility issues with SQLAlchemy (LP: #1834845)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers