Cinder

swift backup driver consistently fails when [DEFAULT] backup_swift_service_auth is set to True

Bug #2058596 reported by Takashi Kajinami on 2024-03-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	Undecided	Takashi Kajinami

Bug Description

When we add

[DEFAULT]
backup_swift_service_auth=True

to cinder.conf to use service user token in swift backup driver, backup operation consistently fails.
The following traceback is found in the log file of cinder-backup service.

```
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session'
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers())
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session'
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server
```

This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1].

[1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736

The specific job using cinder + swift backup driver is puppet-openstack-integration-7-scenario002-tempest-centos-9-stream . The job uses current master packages from RDO + centos 9 stream.

See original description

Tags:

Takashi Kajinami (kajinamit) on 2024-03-21

description:

updated

Takashi Kajinami (kajinamit) on 2024-03-21

description:	updated
tags:	added: backup swift

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-04-16: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/915979

Changed in cinder:
status:	New → In Progress

Takashi Kajinami (kajinamit) on 2024-04-17

Changed in cinder:
assignee:	nobody → Takashi Kajinami (kajinamit)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-18: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/915979
Committed: https://opendev.org/openstack/cinder/commit/9ff29a649ee890cc6bbd7175c8f8f3d2cdd32653
Submitter: "Zuul (22348)"
Branch: master

commit 9ff29a649ee890cc6bbd7175c8f8f3d2cdd32653
Author: Takashi Kajinami <email address hidden>
Date: Tue Apr 16 19:09:00 2024 +0900

Fix broken backup_swift_service_auth=True

    Fix the missing session argument so that swift backup driver to
    resolve the TypeError raised in swift object access. The error makes
    the swift back driver consistently fail during backup/restore
    operation.

Closes-Bug: #2058596
Change-Id: I80f2cd614ba7277a28fa8a4a429fef983113f0fb

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-18: Fix proposed to cinder (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/cinder/+/919867

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-22: Fix merged to cinder (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/919867
Committed: https://opendev.org/openstack/cinder/commit/966bc53c8eab7fc0fdd185bb7305695172a18f58
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 966bc53c8eab7fc0fdd185bb7305695172a18f58
Author: Takashi Kajinami <email address hidden>
Date: Tue Apr 16 19:09:00 2024 +0900

Fix broken backup_swift_service_auth=True

    Closes-Bug: #2058596
    Change-Id: I80f2cd614ba7277a28fa8a4a429fef983113f0fb
    (cherry picked from commit 9ff29a649ee890cc6bbd7175c8f8f3d2cdd32653)