Cinder

  1. Bug #2058596
  2. Activity log

Activity log for bug #2058596

Date Who What changed Old value New value Message
2024-03-21 04:31:17 Takashi Kajinami bug added bug
2024-03-21 04:31:40 Takashi Kajinami description When we configure [DEFAULT] backup_swift_service_auth=True to use service user token in swift backup driver, backup operation consistently fails. The following traceback is found in the log file of cinder-backup service. ``` 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers()) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server ``` This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1]. [1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736 When we add [DEFAULT] backup_swift_service_auth=True to cinder.conf to use service user token in swift backup driver, backup operation consistently fails. The following traceback is found in the log file of cinder-backup service. ``` 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers()) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server ``` This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1]. [1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736
2024-03-21 04:38:45 Takashi Kajinami description When we add [DEFAULT] backup_swift_service_auth=True to cinder.conf to use service user token in swift backup driver, backup operation consistently fails. The following traceback is found in the log file of cinder-backup service. ``` 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers()) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server ``` This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1]. [1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736 When we add [DEFAULT] backup_swift_service_auth=True to cinder.conf to use service user token in swift backup driver, backup operation consistently fails. The following traceback is found in the log file of cinder-backup service. ``` 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server [None req-fdf98bba-85ae-4f96-b505-7953c5073804 c30ae9b4702344f0989538c6c1f8ba17 c2f18cc4e29049bab2e1f2f7cf32e942 - - - -] Exception during message handling: TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 532, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 504, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.force_reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.value 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 494, in continue_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server updates = backup_service.backup( 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 570, in backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server volume_size_bytes) = self._prepare_backup(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 355, in _prepare_backup 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server container = self._create_container(backup) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/chunkeddriver.py", line 258, in _create_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.put_container(backup.container) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 366, in put_container 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server self.conn.head_container(container, headers=self._headers()) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return r(f, *args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise retry_exc.reraise() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self.last_attempt.result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server return self.__get_result() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server raise self._exception 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__ 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/swift.py", line 195, in _headers 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server result['X-Service-Token'] = sa_plugin.get_token() 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server TypeError: get_token() missing 1 required positional argument: 'session' 2024-03-20 10:38:59.637 82432 ERROR oslo_messaging.rpc.server ``` This problem was initially caught in Puppet OpenStack CI when we attempted to enable the feature[1]. [1] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/913736 The specific job using cinder + swift backup driver is puppet-openstack-integration-7-scenario002-tempest-centos-9-stream . The job uses current master packages from RDO + centos 9 stream.
2024-03-21 04:39:37 Takashi Kajinami tags backup swift
2024-04-16 10:11:57 OpenStack Infra cinder: status New In Progress
2024-04-17 01:41:05 Takashi Kajinami cinder: assignee Takashi Kajinami (kajinamit)
2024-05-18 04:48:23 OpenStack Infra cinder: status In Progress Fix Released