Snapshot Backup fails with remotefs based Drivers

Bug #1703405 reported by Silvan Kaiser on 2017-07-10
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Silvan Kaiser
tempest
Undecided
Unassigned

Bug Description

The new tempest test for snapshot backups [1] fails with:

InvalidSnapshot: Snapshot status must be "available" to clone. But is: backing-up

Example log for the Quobyte driver at [2].

As the volume is in the 'backing-up' state the snapshot operation fails as it requires the volume to be in the state 'available'.

This probably hits every Driver that uses the default implementation of _create_volume_from_snapshot from remotefs.py .

[1] https://review.openstack.org/#/c/451170/
[2] http://78.46.57.153:8081/refs-changes-32-482132-1/logs/screen-c-vol.log.txt

Silvan Kaiser (2-silvan) on 2017-07-10
tags: added: quobyte
Eric Harney (eharney) wrote :

The tempest test only checks for "volume_feature_enabled.backup", meaning it equates backup support with snapshot backup support. This is not correct, not all Cinder drivers support snapshot backup.

Change abandoned by Silvan Kaiser (<email address hidden>) on branch: master
Review: https://review.openstack.org/503686
Reason: This just overrides the default and does not fix #1703405

Changed in tempest:
assignee: nobody → Vishakha Agarwal (vishakha.agarwal)
Changed in tempest:
assignee: Vishakha Agarwal (vishakha.agarwal) → nobody

Hi Silvan,

Are you still facing the issue.
I can see mentioned test is currently passing.

Last time, test modified under: https://review.openstack.org/#/c/569917/8/tempest/api/volume/test_volumes_snapshots.py

Seems not reproducible. If still it's issue then could you please provide the reproducing steps?

Uemit Seren (timeu) wrote :

We are running into a similar issue both in above tempest tests as well as when manually creating a backup of a volume with an attached snapshot. This is happening with OSP12 with a Isilon backed NFS cinder backend using the generic NFS driver (The error does not occur when using a NetApp backend with its native driver)

The error in /etc/cinder/backup.log is:

2018-10-11 13:54:30,855.855 785498 ERROR oslo_messaging.rpc.server RemoteError: Remote error: InvalidSnapshot Snapshot status must be "available" to clone. But is: backing-up

The steps to reproduce are:

1.) Create a volume
2.) Create a snapshot of the volume
3.) Create a backup of the volume using the snapshot

openstack volume create test-vol --type nfs --size 1
openstack volume snapshot create --volume test-vol test-snapshot
openstack volume backup create test-vol --snapshot test-snapshot --name test-backup # this fails

If you don't specify the snapshot the backup works:

openstack volume backup create test-vol --name test-backup2 # this works

openstack volume backup list
+--------------------------------------+----------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+----------------+-------------+-----------+------+
| fecd40b9-76da-4fc2-a25c-e38401622068 | test-backup2 | None | available | 1 |
| 169195d5-2939-4f71-9c98-3b5de2db86bb | test-backup | None | error | 1 |
+--------------------------------------+----------------+-------------+-----------+------+

Interestingly the snapshot has the state available:

openstack volume snapshot list
+--------------------------------------+-----------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-----------------+-------------+-----------+------+
| 6310c270-4ec1-4af0-ae9e-cae156d69c98 | test-snapshot | None | available | 1 |
+--------------------------------------+-----------------+-------------+-----------+------+

Silvan Kaiser (2-silvan) wrote :

Hi Manik!
Yes, this issue is still current, i just manually verified to be sure.
Running current master branches with Devstack and using the Cinder Quobyte driver and the PosixBackupDriver to run the tempest test referenced above.

I think Eric pointed out the core issue, not all Cinder drivers support backup by snapshot, this might require another config flag?

Ghanshyam Mann (ghanshyammann) wrote :

yeah as per Eric, not all cinder driver support snapshot backup. is there any way to detect this via cinder API?

I am hesitate to introduce the feature flag for this feature. if this is commonly implemented/supported feature then, we can ask driver to implement it and till then they can skip this test to run maually via backlist.

Jay Bryant (jsbryant) wrote :

There are a few features that fail tempest testing on various backends. They blacklist those tests so that testing passes. I don't know off the top of my head how many drivers don't implement this but I think that just disabling the test for drivers that don't is the best way to go rather than trying to add more config options.

Ghanshyam Mann (ghanshyammann) wrote :

I agree with Jay. marking invalid for Tempest.

Changed in tempest:
status: New → Invalid

Fix proposed to branch: master
Review: https://review.opendev.org/683103

Changed in cinder:
assignee: nobody → Silvan Kaiser (2-silvan)
status: New → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers