OpenStack Cinder-backup Charm

Backup volumes in error state

Bug #1897587 reported by Aurelien Lourot on 2020-09-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Cinder-backup Charm	Invalid	High	Unassigned

Bug Description

Happened once on OSCI (bionic-train):

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-cinder-backup/748623/6/19381/index.html

Can't reproduce locally, so it must be quite rare.

2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server [req-336a3d45-b494-4134-970b-6d8b4c9759f7 7797bffafd754935b6f67eb6e3883728 71eec3a4979940ceb079b672a09fa65b - 25706a6ee8cc43b9aa2ad81e5f6c34eb 25706a6ee8cc43b9aa2ad81e5f6c34eb] Exception during message handling: rados.ObjectNotFound: [errno 2] error opening pool 'b'zaza-cinderbackuptests-410-vol-1-backup''
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 417, in create_backup
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server raise value
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 406, in create_backup
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 494, in _run_backup
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path))
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 973, in backup
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server length)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 740, in _backup_rbd
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server length)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 681, in _full_rbd_backup
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server container)) as client:
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/rbd.py", line 195, in __init__
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 305, in _connect_to_rados
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server ioctx = client.open_ioctx(pool_to_open)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 190, in doit
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 148, in proxy_call
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 129, in execute
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server raise value
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "rados.pyx", line 516, in rados.requires.wrapper.validate_func
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server File "rados.pyx", line 1203, in rados.Rados.open_ioctx
2020-09-28 10:27:11.829 3138 ERROR oslo_messaging.rpc.server rados.ObjectNotFound: [errno 2] error opening pool 'b'zaza-cinderbackuptests-410-vol-1-backup''

Not a duplicate of #1877076 because in #1877076 the new backup volume remains forever in "creating" state whereas here the new backup volume ends up in error state.