Cinder

  1. Bug #1764269
  2. Comment #2

Comment 2 for bug 1764269

Revision history for this message
Eric Miller (erickmiller) wrote :

We're using Kolla-Ansible deployed Rocky (7.0.2) with Swift as the backup driver. We use Swift emulation on Ceph, not native Swift. Backups work fine, so communication with the Swift emulation layer is working. Backups are stored in the respective project's volumebackups container (chunks, metadata, and sha256file files).

However, I have seen this same issue where deleting volume backups using "openstack volume backup delete <uuid>" results in a successful deletion of the volume backup record, but the objects in Swift are never deleted. There is no error logged in any of the cinder logs on any node (controllers nor compute), only a successful deletion record in the cinder-backup.log file on the compute node. The debug flag is set to "True" in all config files.

I included the cinder-backup.log entries below. This is indicative of the issue - the object list is
generated object list: []

even though running this, as one of the project users, results in a list of all of the backup objects (152 of them, including the chunks, metadata, and sha256file files):
swift list -p volume_01f54377-b694-408c-a64e-54650f076900/20190827075627/az_us-central-1a_backup_51e2d6a1-b171-46cb-96cc-9786a684661b volumebackups

I haven't tried installing native Swift, since it is unsupported alongside Ceph in Kolla Ansible, so can't try to run the swift proxy service to see if it solves the issue.

At this point, we are thinking of going back to using Ceph native for backups, unless someone has an idea why this is happening.

Eric

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.131 6 INFO cinder.backup.manager [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] Delete backup started, backup: 51e2d6a1-b171-46cb-96cc-9786a684661b.

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.137 6 DEBUG cinder.backup.drivers.swift [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] Using swift URL http://192.168.1.254:6780/swift/v1/AUTH_8acb2072057b45ab9da245880af92c93 initialize /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/swift.py:248

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.138 6 DEBUG cinder.backup.drivers.swift [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] Connect to http://192.168.1.254:6780/swift/v1/AUTH_ in "per_user" mode initialize /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/drivers/swift.py:250

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.138 6 DEBUG cinder.backup.chunkeddriver [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] delete started, backup: 51e2d6a1-b171-46cb-96cc-9786a684661b, container: volumebackups, prefix: volume_01f54377-b694-408c-a64e-54650f076900/20190827075627/az_us-central-1a_backup_51e2d6a1-b171-46cb-96cc-9786a684661b. delete_backup /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py:807

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.193 6 DEBUG cinder.backup.chunkeddriver [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] generated object list: []. _generate_object_names /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py:232

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.193 6 DEBUG cinder.backup.chunkeddriver [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] delete 51e2d6a1-b171-46cb-96cc-9786a684661b finished. delete_backup /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py:829

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.199 6 DEBUG oslo_db.sqlalchemy.engines [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:308

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.215 6 DEBUG cinder.quota [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] Created reservations ['37abb9f4-a87d-4a94-8458-18e912ca0e12', '0d21ec1c-dd5a-4096-8b0e-9a09f889bf5e'] reserve /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/quota.py:1029

/var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-backup.log:2019-09-30 13:46:53.229 6 INFO cinder.backup.manager [req-c8187c1a-34be-435b-9200-00f948699899 720763d70a094bf3b11ebabd31eee896 8acb2072057b45ab9da245880af92c93 - default default] Delete backup finished, backup 51e2d6a1-b171-46cb-96cc-9786a684661b deleted.