Backups with remotefs hit AttributeError for field '_context'

Bug #1744692 reported by Silvan Kaiser
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Silvan Kaiser

Bug Description

When running volume backups with remotefs based drivers that use snapshots the volume cloning fails with the trace (Quobyte driver example):

Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server [None req-ee3d24fc-8f16-42cc-8fd7-84f7cd9b4ec4 tempest-VolumesBackupsV39Test-1006542993 None] Exception during message handling:
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4291, in get_backup_device
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server self.driver.get_backup_device(ctxt, backup))
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 1141, in get_backup_device
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server context, backup)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 1182, in _get_backup_volume_temp_volume
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server context, volume)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 1287, in _create_temp_cloned_volume
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server temp_vol_ref.destroy()
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/driver.py", line 1282, in _create_temp_cloned_volume
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server model_update = self.create_cloned_volume(temp_vol_ref, volume)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/quobyte.py", line 190, in create_cloned_volume
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server return self._create_cloned_volume(volume, src_vref)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1038, in _create_cloned_volume
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server self._create_snapshot(temp_snapshot)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1423, in _create_snapshot
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server new_snap_path)
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1436, in _create_snapshot_online
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server context = snapshot._context
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server AttributeError: 'Snapshot' object has no attribute '_context'
Jan 22 09:56:27 manualcinderci cinder-volume[997]: ERROR oslo_messaging.rpc.server

This occurs with the Quobyte driver but the code run is almost exclusively from the remotefs driver classes, which is why is think this will be hit by most remotefs drivers.
It can be reproduced by running the Tempest test_volumes_backup test
tempest.api.volume.test_volumes_backup.VolumesBackupsTest.test_backup_create_attached_volume
with a remotefs based driver that uses snapshots for cloning volumes.

Revision history for this message
livelace (k-a2min-2) wrote :

I have got this problem too. NFS storage, Queens, openstack-cinder-12.0.1-1.el7.noarch

Revision history for this message
Silvan Kaiser (2-silvan) wrote :

@livelace: Not sure it makes much of a diff but you might add your me too to the bug heat tracker (top of page, fire icon)

Revision history for this message
TommyLike (hu-husheng) wrote :

By searching the cinder code, I found the snapshot object that is being used in the method is a temporary object made via namedtuple, and that means the OVO attribute '_context' is missing since it's made [1]. I don't have an environment by hand, not sure this change can fix the error.

```
snap_attrs = ['volume_name', 'volume_size', 'name',
                          'volume_id', 'id', 'volume', '_context']
            Snapshot = collections.namedtuple('Snapshot', snap_attrs)
            temp_snapshot = Snapshot(volume_name=volume_name,
                                     volume_size=src_vref.size,
                                     name='clone-snap-%s' % src_vref.id,
                                     volume_id=src_vref.id,
                                     id='tmp-snap-%s' % src_vref.id,
                                     volume=src_vref,
                                     _context=src_vref._context)
```

[1]: https://github.com/openstack/cinder/blob/5dcf4f52ada36b98d08584cd31a55698006d9e35/cinder/volume/drivers/remotefs.py#L1044

Revision history for this message
Silvan Kaiser (2-silvan) wrote :

@tommylike: I did a quick&dirty test run, collections stumbled over ValueError: Field names cannot start with an underscore: '_context' .
I should be able to take a closer look at this tomorrow...

Revision history for this message
TommyLike (hu-husheng) wrote :

Thanks, @Silvan Kaiser, if that so, we still can get the ``_context`` attribute via snapshot.volume._context。

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/570885

Changed in cinder:
assignee: nobody → Silvan Kaiser (2-silvan)
status: New → In Progress
Revision history for this message
Silvan Kaiser (2-silvan) wrote :

@tommylike: I created a change at https://review.openstack.org/#/c/570885/ for bringing this forward. Please see my comments in that change.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/570885
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=78f2101fc5a84e81132e6c2adc89cfe681018d77
Submitter: Zuul
Branch: master

commit 78f2101fc5a84e81132e6c2adc89cfe681018d77
Author: Silvan Kaiser <email address hidden>
Date: Tue May 29 12:17:43 2018 +0200

    Add context to cloning snapshots in remotefs driver

    Adds context to _create_cloned_volume method in order
    to allow taking a snapshot during cloning. Also saves
    the temp_snapshot in order to enable Nova to update the
    snapshot state. Destroys the temp_snapshot after usage.
    This affects the NFS, WindowsSMBFS, VZStorage and
    Quobyte drivers.

    Closes-Bug: #1744692

    Change-Id: I328a02d0f26e8c3d41ec18a0487da6fd20b39b04

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/681515

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/stein)

Reviewed: https://review.opendev.org/681515
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=247c6607156da6f4403c4826c7d2e09720c43e8d
Submitter: Zuul
Branch: stable/stein

commit 247c6607156da6f4403c4826c7d2e09720c43e8d
Author: Silvan Kaiser <email address hidden>
Date: Tue May 29 12:17:43 2018 +0200

    Add context to cloning snapshots in remotefs driver

    Adds context to _create_cloned_volume method in order
    to allow taking a snapshot during cloning. Also saves
    the temp_snapshot in order to enable Nova to update the
    snapshot state. Destroys the temp_snapshot after usage.
    This affects the NFS, WindowsSMBFS, VZStorage and
    Quobyte drivers.

    Closes-Bug: #1744692

    Change-Id: I328a02d0f26e8c3d41ec18a0487da6fd20b39b04
    (cherry picked from commit 78f2101fc5a84e81132e6c2adc89cfe681018d77)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/682097

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 14.0.2

This issue was fixed in the openstack/cinder 14.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/rocky)

Reviewed: https://review.opendev.org/682097
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b886d093d782301829afb42e6476b0f5a4678fba
Submitter: Zuul
Branch: stable/rocky

commit b886d093d782301829afb42e6476b0f5a4678fba
Author: Silvan Kaiser <email address hidden>
Date: Tue May 29 12:17:43 2018 +0200

    Add context to cloning snapshots in remotefs driver

    Adds context to _create_cloned_volume method in order
    to allow taking a snapshot during cloning. Also saves
    the temp_snapshot in order to enable Nova to update the
    snapshot state. Destroys the temp_snapshot after usage.
    This affects the NFS, WindowsSMBFS, VZStorage and
    Quobyte drivers.

    Closes-Bug: #1744692

    Change-Id: I328a02d0f26e8c3d41ec18a0487da6fd20b39b04
    (cherry picked from commit 78f2101fc5a84e81132e6c2adc89cfe681018d77)
    (cherry picked from commit 247c6607156da6f4403c4826c7d2e09720c43e8d)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/688131

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/queens)

Reviewed: https://review.opendev.org/688131
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=772897cad777716e55216f59d0e11ec191b31c80
Submitter: Zuul
Branch: stable/queens

commit 772897cad777716e55216f59d0e11ec191b31c80
Author: Silvan Kaiser <email address hidden>
Date: Tue May 29 12:17:43 2018 +0200

    Add context to cloning snapshots in remotefs driver

    Adds context to _create_cloned_volume method in order
    to allow taking a snapshot during cloning. Also saves
    the temp_snapshot in order to enable Nova to update the
    snapshot state. Destroys the temp_snapshot after usage.
    This affects the NFS, WindowsSMBFS, VZStorage and
    Quobyte drivers.

    Closes-Bug: #1744692

    Change-Id: I328a02d0f26e8c3d41ec18a0487da6fd20b39b04
    (cherry picked from commit 78f2101fc5a84e81132e6c2adc89cfe681018d77)
    (cherry picked from commit 247c6607156da6f4403c4826c7d2e09720c43e8d)
    (cherry picked from commit b886d093d782301829afb42e6476b0f5a4678fba)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.10

This issue was fixed in the openstack/cinder 12.0.10 release.

Revision history for this message
David Hill (david-hill-ubisoft) wrote :
Download full text (5.1 KiB)

This appears to be breaking boot from volume:

~~~
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server [req-fce84ebf-1a56-4739-bc1b-314e8339c53c 2708e4b6a36d4c61b96363e07d7f87e0 1445bbc727c54d1bab30d308c340eded - default default] Exception during message handling: AttributeError: 'Volume' object has no attribute 'obj_context'
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 690, in create_volume
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server _run_flow()
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 682, in _run_flow
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server flow_engine.run()
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server failures[0].reraise()
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2019-11-27 18:14:32.328 77 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", li...

Read more...

Revision history for this message
Eric Harney (eharney) wrote :

I think comment #18 comes from a downstream patched up version of queens carrying patches that aren't on upstream queens. The same failure may or may not reproduce on upstream Cinder.

Please include this context if adding stack traces to LP bugs.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 13.0.8

This issue was fixed in the openstack/cinder 13.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.