Snapshot Backup fails with remotefs based Drivers

[Silvan Kaiser]

The new tempest test for snapshot backups [1] fails with:

InvalidSnapshot: Snapshot status must be "available" to clone. But is: backing-up

Example log for the Quobyte driver at [2].

As the volume is in the 'backing-up' state the snapshot operation fails as it requires the volume to be in the state 'available'.

This probably hits every Driver that uses the default implementation of _create_volume_from_snapshot from remotefs.py .

[1] https://review.openstack.org/#/c/451170/
[2] http://78.46.57.153:8081/refs-changes-32-482132-1/logs/screen-c-vol.log.txt

[Eric Harney (eharney)]

The tempest test only checks for "volume_feature_enabled.backup", meaning it equates backup support with snapshot backup support. This is not correct, not all Cinder drivers support snapshot backup.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/503686

[OpenStack Infra (hudson-openstack)]

Change abandoned by Silvan Kaiser (<email address hidden>) on branch: master
Review: https://review.openstack.org/503686
Reason: This just overrides the default and does not fix #1703405

[Manik Bindlish (manikbindlish19)]

Hi Silvan,

Are you still facing the issue.
I can see mentioned test is currently passing.

Last time, test modified under: https://review.openstack.org/#/c/569917/8/tempest/api/volume/test_volumes_snapshots.py

Seems not reproducible. If still it's issue then could you please provide the reproducing steps?

[Uemit Seren (timeu)]

We are running into a similar issue both in above tempest tests as well as when manually creating a backup of a volume with an attached snapshot. This is happening with OSP12 with a Isilon backed NFS cinder backend using the generic NFS driver (The error does not occur when using a NetApp backend with its native driver)

The error in /etc/cinder/backup.log is:

2018-10-11 13:54:30,855.855 785498 ERROR oslo_messaging.rpc.server RemoteError: Remote error: InvalidSnapshot Snapshot status must be "available" to clone. But is: backing-up

The steps to reproduce are:

1.) Create a volume
2.) Create a snapshot of the volume
3.) Create a backup of the volume using the snapshot

openstack volume create test-vol --type nfs --size 1
openstack volume snapshot create --volume test-vol test-snapshot
openstack volume backup create test-vol --snapshot test-snapshot --name test-backup # this fails

If you don't specify the snapshot the backup works:

openstack volume backup create test-vol --name test-backup2 # this works

openstack volume backup list
+--------------------------------------+----------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+----------------+-------------+-----------+------+
| fecd40b9-76da-4fc2-a25c-e38401622068 | test-backup2 | None | available | 1 |
| 169195d5-2939-4f71-9c98-3b5de2db86bb | test-backup | None | error | 1 |
+--------------------------------------+----------------+-------------+-----------+------+

Interestingly the snapshot has the state available:

openstack volume snapshot list
+--------------------------------------+-----------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-----------------+-------------+-----------+------+
| 6310c270-4ec1-4af0-ae9e-cae156d69c98 | test-snapshot | None | available | 1 |
+--------------------------------------+-----------------+-------------+-----------+------+

[Silvan Kaiser (2-silvan)]

Hi Manik!
Yes, this issue is still current, i just manually verified to be sure.
Running current master branches with Devstack and using the Cinder Quobyte driver and the PosixBackupDriver to run the tempest test referenced above.

I think Eric pointed out the core issue, not all Cinder drivers support backup by snapshot, this might require another config flag?

[Ghanshyam Mann (ghanshyammann)]

yeah as per Eric, not all cinder driver support snapshot backup. is there any way to detect this via cinder API?

I am hesitate to introduce the feature flag for this feature. if this is commonly implemented/supported feature then, we can ask driver to implement it and till then they can skip this test to run maually via backlist.

[Jay Bryant (jsbryant)]

There are a few features that fail tempest testing on various backends. They blacklist those tests so that testing passes. I don't know off the top of my head how many drivers don't implement this but I think that just disabling the test for drivers that don't is the best way to go rather than trying to add more config options.

[Ghanshyam Mann (ghanshyammann)]

I agree with Jay. marking invalid for Tempest.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/683103

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/683103
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=06b9e3c498b833aa602cb10ca640979850af517a
Submitter: Zuul
Branch: master

commit 06b9e3c498b833aa602cb10ca640979850af517a
Author: Silvan Kaiser <email address hidden>
Date: Thu Sep 19 12:11:02 2019 +0200

    Allow creating volumes from snapshots during backups for Quobyte

    Allows to create a volume from a snapshot while the snapshot itself is in
    the state 'backing-up'. Moves the corresponding unit tests to ddt to
    extend coverage while reducing code size.

    Partial-Bug: #1703405

    Change-Id: If4fe96ac59c975ba71c4387004d7454cb293b972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722353
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=406e0778ec0be8455091684667478d7a570ceaa0
Submitter: Zuul
Branch: master

commit 406e0778ec0be8455091684667478d7a570ceaa0
Author: Silvan Kaiser <email address hidden>
Date: Thu Apr 23 17:11:32 2020 +0200

    Nit: Update comment and release note on Quobyte driver bugfix

    Updates the version description to be more accurate and the
    release note to contain the bug ticket reference.

    Partial-bug: #1703405

    Change-Id: Ie1e2d635434f14adbc0b67c1ec2583842e1b4f6d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/724690

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724690
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=68b381bf252939b48bbcc1e3e4c279ba7233d1d0
Submitter: Zuul
Branch: stable/ussuri

commit 68b381bf252939b48bbcc1e3e4c279ba7233d1d0
Author: Silvan Kaiser <email address hidden>
Date: Thu Apr 23 17:11:32 2020 +0200

    Nit: Update comment and release note on Quobyte driver bugfix

    Updates the version description to be more accurate and the
    release note to contain the bug ticket reference.

    Partial-bug: #1703405

    Change-Id: Ie1e2d635434f14adbc0b67c1ec2583842e1b4f6d
    (cherry picked from commit 406e0778ec0be8455091684667478d7a570ceaa0)