Cinder backup failed for restore volume snapshot.
Works for restore volumes without snapshots.
I use a external ceph cluster version 14.2.6 nautilus (stable)
What you expected to happen:
Restore volume snapshot backup.
How to reproduce it (minimal and precise):
- Create a instance with disk;
- Snaptshot instance;
- Backup volume snapshot;
Example: openstack volume backup create 18363c93-e577-470e-971a-678094bce943 --snapshot 1624b162-e616-47dc-bb1b-01fe4843c581
- Restore backup;
CentOS Linux release 7.7.1908 (Core)
3.10.0-1062.9.1.el7.x86_64
Docker 19.03.5
Docker image source
Docker image distribution Centos
official images from Docker Hub
git status: # On branch stable/train
==> /var/log/kolla/cinder/cinder-backup.log <==
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server [req-d37e8900-4c9b-45d7-9b6b-59dd66159e86 0af9f5deabf74f4e8efea750cb3c7cf0 1baa170cb3a742e59db2e7e5948d6e7f - default default] Exception during message handling: ImageNotFound: [errno 2] error opening image volume-18363c93-e577-470e-971a-678094bce943.backup.base at snapshot None
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 608, in restore_backup
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server backup.save()
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 597, in restore_backup
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server self._run_restore(context, backup, volume)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 659, in _run_restore
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path))
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1264, in restore
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server self._restore_volume(backup, target_volume, volume_file)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1231, in _restore_volume
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server length, src_snap=restore_point)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/ceph.py", line 1026, in _full_restore
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server read_only=True))
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server File "rbd.pyx", line 2753, in rbd.Image.__init__ (/builddir/build/BUILD/ceph-14.2.1/build/src/pybind/rbd/pyrex/rbd.c:24354)
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server ImageNotFound: [errno 2] error opening image volume-18363c93-e577-470e-971a-678094bce943.backup.base at snapshot None
2020-01-23 22:10:39.340 29 ERROR oslo_messaging.rpc.server
# cat /etc/kolla/config/cinder/cinder-volume.conf
[DEFAULT]
enabled_backends=netapp06,rbd-1
debug = True
[rbd-1]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=volumes
volume_backend_name=rbd-1
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
[netapp06]
volume_backend_name=netapp06
nfs_shares_config=/etc/cinder/nfs_shares
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_snapshot_support = True
nas_secure_file_operations = False
image_volume_cache_enabled = True
# cat /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder-backup.keyring
[client.cinder-backup]
key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(kolla) [root@kolla-deploy config]# cat /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
[client.cinder]
key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(kolla) [root@kolla-deploy config]#
# cat /etc/kolla/config/cinder/cinder-backup.conf
[DEFAULT]
backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user=cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_pool=backups
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true
(kolla) [root@kolla-deploy config]#
# cat /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
[client.cinder]
key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# cat /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring
[client.cinder-backup]
key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(kolla) [root@kolla-deploy config]#
# grep ^[^#] /etc/kolla/globals.yml
---
kolla_install_type: "source"
openstack_release: "train"
kolla_internal_vip_address: "172.16.120.154"
kolla_external_vip_address: "XXX.XXX.XXX.XXX"
kolla_external_fqdn: "cloud.xxxxxx.com.br"
kolla_enable_tls_external: "yes"
kolla_external_fqdn_cert: "{{ node_config }}/certificates/xxxxxx.pem"
kolla_external_fqdn_cacert: "{{ node_config }}/certificates/xxxxxx-ca.crt"
enable_cinder: "yes"
enable_cinder_backup: "yes"
enable_mariabackup: "yes"
enable_neutron_vpnaas: "yes"
enable_neutron_agent_ha: "yes"
glance_backend_ceph: "yes"
glance_backend_file: "no"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"
Informing cinder - they might know how to approach the issue.