Cinder

swift backup driver consistently fails when [DEFAULT] backup_swift_service_auth is set to True

Bug #2058596 reported by Takashi Kajinami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Undecided
Takashi Kajinami

Bug Description

When we add

[DEFAULT]
backup_swift_service_auth=True

to cinder.conf to use service user token in swift backup driver, backup operation consistently fails.
The following traceback is found in the log file of cinder-backup service.

```
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session'
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers())
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs)
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token()
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session'
2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server
```

This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1].

[1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736

The specific job using cinder + swift backup driver is puppet-openstack-integration-7-scenario002-tempest-centos-9-stream . The job uses current master packages from RDO + centos 9 stream.

See original description
Tags: backup swift
Takashi Kajinami (kajinamit)
description: updated
Takashi Kajinami (kajinamit)
description: updated
tags: added: backup swift
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/915979

Changed in cinder:
status: New → In Progress
Takashi Kajinami (kajinamit)
Changed in cinder:
assignee: nobody → Takashi Kajinami (kajinamit)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.