online snapshot deletion breaks backing chain with remotefs drivers
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Invalid
|
High
|
Unassigned | ||
OpenStack Compute (nova) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
The deletion of online snapshots of remotefs based volumes breaks the .info file/backing chain of these volumes. Logs can be seen in any current Quobyte CI run in Cinder/
I ran a lot of tests and so far i can say that the first deletion of a member in the backing chain works (snapshot is deleted) but seemingly leaves the .info files content and/or the backing chain of the volume file in a broken state. The error can be identified e.g. by the following log pattern:
This is the first snapshot deletion that runs successfully (the snapshots id is 91755e5f-
2017-09-13 08:28:59.436 20467 DEBUG cinder.
fc89 of volume 94598844-
2017-09-13 08:28:59.487 20467 DEBUG cinder.
The next snapshot to be deleted (138a1f62-
2017-09-13 08:29:01.857 20467 DEBUG cinder.
eeaf of volume 94598844-
2017-09-13 08:29:01.872 20467 DEBUG cinder.
Now this second snapshot deletion fails because the snapshot file for 138a1f62-
2017-09-13 08:29:02.674 20467 ERROR oslo_messaging.
2017-09-13 08:29:02.674 20467 ERROR oslo_messaging.
2017-09-13 08:29:02.674 20467 ERROR oslo_messaging.
2017-09-13 08:29:02.674 20467 ERROR oslo_messaging.
2017-09-13 08:29:02.674 20467 ERROR oslo_messaging.
The referenced tempest test fails 100% of the time in our CIs. I manually tested the scenario and found the same results. Furthermore i was able, by creating three consecutive snapshots from a single volume and deleting them one after the other, to create a snapshot file with a broken backing file link. In the end i was left with a volume file and an overlay file referencing a removed backing file (previous snapshot of the same volume).
I was able to run the scenario without issues when using offline snapshots. Thus this seems to be related to the usage of the online snapshot deletion via the Nova API.
tags: | added: quobyte remotefs |
description: | updated |
tags: | added: nfs |
summary: |
- online snapshot deletion breaks volume info and backing chain (with - remotefs drivers?) + online snapshot deletion breaks backing chain with remotefs drivers |
Changed in cinder: | |
importance: | Undecided → High |
tags: | added: bugsmash |
Changed in nova: | |
status: | Expired → Invalid |
This issue started up around the 28th of August but definitely not earlier than the 24th of August.