Rbd: possibility of data loss during revert-to-snap

[Margarita Shakhova]

Recently implemented rever-to-snapshot function for RBD driver does not take into account if the rbd image which corresponds to volume is actually busy. This may happen for example if the volume status was resetted without actual detach.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/826036

[Sofia Enriquez (lsofia-enriquez)]

Greetings Margarita Shakhova,
Would you mind sharing the next information?
- Steps to reproduce the error
- Cinder logs with the problem (c-vol DEBUG logs) or logs that show the data loss.
- Are you using master branch or other release?
Thanks in advance

[Margarita Shakhova (shakhova-margarita)]

Greetings Sofia Enriquez,

1.Steps are for example the following:

- Boot an instance from a volume (ceph backend)
- Create a snapshot of the volume
- Reset state of this volume to available
- Try to revert to the latest snapshot.

Alternatively, there could be some delays/problems with cinder services, e.g. timeouts which could result to the volume which can be reverted from the point of view of the cinder API (state was reset to available), but such revert would destroy data.

2. Un fact there are no problems in the Cinder logs itself. All problems are located within backend. The source of the problem is that it is unacceptable to rollback image with watchers while there are not any check if the watchers are present.

I will try to find ceph logs indicating this problem.

[Margarita Shakhova (shakhova-margarita)]

This log message is for a boot volume (created from image) which is actually attached and than reset to available:

2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager [req-3c679b45-ab7d-4fe2-bded-ba7db533bc0d 91c84696756645daa02de7e70def58fb e2a9a87d227647d481255205c42c5c80 - default default] Volume d31c87da-82e8-4f96-887e-74dac56b7dc3 revert to snapshot c7db0d01-b9b4-4f71-ba21-7c0784c2549a failed with [errno 30] error rolling back image b'volume-d31c87da-82e8-4f96-887e-74dac56b7dc3' to snapshot b'snapshot-c7db0d01-b9b4-4f71-ba21-7c0784c2549a'.: rbd.ReadOnlyImage: [errno 30] error rolling back image b'volume-d31c87da-82e8-4f96-887e-74dac56b7dc3' to snapshot b'snapshot-c7db0d01-b9b4-4f71-ba21-7c0784c2549a'
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager Traceback (most recent call last):
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/manager.py", line 1063, in revert_to_snapshot
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager self._revert_to_snapshot(context, volume, snapshot)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/manager.py", line 1011, in _revert_to_snapshot
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager self.driver.revert_to_snapshot(context, volume, snapshot)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 1294, in revert_to_snapshot
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager image.rollback_to_snap(snapshot.name)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager rv = execute(f, *args, **kwargs)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager six.reraise(c, e, tb)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager raise value
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager rv = meth(*args, **kwargs)
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "rbd.pyx", line 2882, in rbd.requires_not_closed.wrapper
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager File "rbd.pyx", line 3621, in rbd.Image.rollback_to_snap
2022-02-09 21:20:58.720 31 ERROR cinder.volume.manager rbd.ReadOnlyImage: [errno 30] error rolling back image b'volume-d31c87da-82e8-4...