Ubuntu
python-os-brick package

  1. Bug #1865754
  2. Activity log

Activity log for bug #1865754

Date Who What changed Old value New value Message
2020-03-03 03:09:34 wes hayutin bug added bug
2020-03-03 03:09:55 wes hayutin summary tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ due to containers going down tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument
2020-03-03 05:30:46 chandan kumar tags promotion-blocker alert promotion-blocker tempest
2020-03-09 22:39:22 wes hayutin tripleo: status Triaged Fix Released
2020-05-21 17:47:24 Corey Bryant bug task added os-brick
2020-05-22 15:25:08 Corey Bryant bug task added python-os-brick (Ubuntu)
2020-05-22 15:25:19 Corey Bryant python-os-brick (Ubuntu): status New Triaged
2020-05-22 15:25:31 Corey Bryant python-os-brick (Ubuntu): importance Undecided Critical
2020-08-14 10:01:11 Alex Kavanagh bug task added charm-ceph-rbd-mirror
2020-09-02 14:14:42 OpenStack Infra os-brick: status In Progress Fix Released
2020-09-09 18:21:25 OpenStack Infra tags alert promotion-blocker tempest alert in-stable-victoria promotion-blocker tempest
2020-09-14 10:54:51 OpenStack Infra tags alert in-stable-victoria promotion-blocker tempest alert in-stable-ussuri in-stable-victoria promotion-blocker tempest
2020-09-24 09:47:15 Launchpad Janitor merge proposal linked https://code.launchpad.net/~chris.macnaughton/ubuntu/+source/python-os-brick/+git/python-os-brick/+merge/391267
2020-09-25 18:13:49 Corey Bryant nominated for series Ubuntu Focal
2020-09-25 18:13:49 Corey Bryant bug task added python-os-brick (Ubuntu Focal)
2020-09-25 18:13:49 Corey Bryant nominated for series Ubuntu Groovy
2020-09-25 18:13:49 Corey Bryant bug task added python-os-brick (Ubuntu Groovy)
2020-09-25 18:14:07 Corey Bryant python-os-brick (Ubuntu Focal): status New Triaged
2020-09-25 18:14:10 Corey Bryant python-os-brick (Ubuntu Focal): importance Undecided Critical
2020-09-25 18:14:17 Corey Bryant python-os-brick (Ubuntu Groovy): status Triaged Fix Released
2020-09-25 18:14:46 Corey Bryant bug task added cloud-archive
2020-09-25 18:14:59 Corey Bryant nominated for series cloud-archive/ussuri
2020-09-25 18:14:59 Corey Bryant bug task added cloud-archive/ussuri
2020-09-25 18:14:59 Corey Bryant nominated for series cloud-archive/victoria
2020-09-25 18:14:59 Corey Bryant bug task added cloud-archive/victoria
2020-09-25 18:15:06 Corey Bryant cloud-archive/ussuri: status New Incomplete
2020-09-25 18:15:09 Corey Bryant cloud-archive/ussuri: status Incomplete Triaged
2020-09-25 18:15:15 Corey Bryant cloud-archive/ussuri: importance Undecided Critical
2020-09-25 18:15:26 Corey Bryant cloud-archive/victoria: status New Fix Released
2020-09-28 06:54:32 Chris MacNaughton summary tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument [SRU] tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument
2020-09-28 07:07:08 Chris MacNaughton description https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/job-output.txt 2020-03-02 18:07:56.142233 | primary | 2020-03-02 18:07:52.442 135023 WARNING urllib3.connectionpool [-] Retrying (Retry(total=8, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))': /v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file 2020-03-02 18:07:56.142265 | primary | 2020-03-02 18:07:55.915 135023 INFO tempest.lib.common.rest_client [req-2c4e9f5f-5497-4eae-b26c-9203f8fbc85b ] Request (main): 500 PUT http://192.168.24.1:9292/v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file 14.582s 2020-03-02 18:07:56.142297 | primary | 2020-03-02 18:07:55.917 135023 CRITICAL tempest [-] Unhandled error: tempest.lib.exceptions.ServerFault: Got server fault 2020-03-02 18:07:56.142328 | primary | Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n ' 2020-03-02 18:07:56.142377 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Traceback (most recent call last): 2020-03-02 18:07:56.142411 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/bin/discover-tempest-config", line 10, in <module> 2020-03-02 18:07:56.142442 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest sys.exit(main()) 2020-03-02 18:07:56.142473 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 605, in main 2020-03-02 18:07:56.142505 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest verbose=args.verbose 2020-03-02 18:07:56.142549 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 545, in config_tempest 2020-03-02 18:07:56.142582 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image.create_tempest_images(conf) 2020-03-02 18:07:56.142612 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 113, in create_tempest_images 2020-03-02 18:07:56.142644 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image_dest=img_path) 2020-03-02 18:07:56.142675 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 161, in find_or_upload_image 2020-03-02 18:07:56.142706 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image = self._upload_image(image_name, image_dest) 2020-03-02 18:07:56.142737 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 212, in _upload_image 2020-03-02 18:07:56.142769 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self.client.store_image_file(image['id'], data) 2020-03-02 18:07:56.142801 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 151, in store_image_file 2020-03-02 18:07:56.142832 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest body=data, chunked=True) 2020-03-02 18:07:56.142863 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request 2020-03-02 18:07:56.142894 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self._error_checker(resp, resp_body) 2020-03-02 18:07:56.142926 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 864, in _error_checker 2020-03-02 18:07:56.142965 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest message=message) 2020-03-02 18:07:56.142996 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest tempest.lib.exceptions.ServerFault: Got server fault 2020-03-02 18:07:56.143049 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n ' 2020-03-02 18:07:56.143083 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest  2020-03-02 18:07:56.143113 | primary | 2020-03-02 18:07:56.143144 | primary | 2020-03-02 18:07:56.143192 | primary | STDERR: 2020-03-02 18:07:56.143225 | primary | 2020-03-02 18:07:56.143256 | primary | /usr/lib/python3.6/site-packages/config_tempest/profile.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. 2020-03-02 18:07:56.143286 | primary | return yaml.load(stream) ===================================== Actual error: is here: https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/logs/undercloud/var/log/containers/glance/api.log InvalidArgumentError('RADOS invalid argument (error calling conf_read_file)',) trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:156 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Exception while accessing to cinder volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data']) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder result = f(*args, **kwargs) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_cluster_name=str(cluster_name)) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__ 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder self.client, self.ioctx = self.connect() 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder conffile=self.rbd_conf) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 692, in rados.Rados.__init__ 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 743, in rados.Rados.__setup 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 826, in rados.Rados.conf_read_file 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder 2020-03-02 18:07:52.296 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to write to volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.386 40 ERROR glance.api.v2.image_data [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to upload image data due to internal error: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Caught error: [errno 22] RADOS invalid argument (error calling conf_read_file): rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi Traceback (most recent call last): 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi request, **action_args) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1390, in dispatch 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return method(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 415, in wrapped 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 299, in upload 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._restore(image_repo, image) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 164, in upload 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 501, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 448, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 204, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 319, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 548, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 461, in _upload_to_store 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi verifier=verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 396, in add_with_multihash 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 478, in store_add_to_backend_with_multihash 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 747, in add 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return next(self.gen) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi device = conn.connect_volume(connection_info['data']) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi result = f(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_handle = self._get_rbd_handle(connection_properties) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_cluster_name=str(cluster_name)) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.client, self.ioctx = self.connect() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi conffile=self.rbd_conf) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 692, in rados.Rados.__init__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 743, in rados.Rados.__setup 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 826, in rados.Rados.conf_read_file 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) [Impact] Os-brick is unusable with Ceph Octopus because of a tightening of config formatting requirements in Ceph. [Test Case] Validate cinder-backup with Ceph on Ubuntu Focal, in addition to testing with Ceph on Xenial (OpenStack Mitaka) with Cinder (and cinder-backup) on Focal (OpenStack Ussuri) [Regression Potential] The risk should be fairly low as the new confiig line being generated is supported since Ceph Hammer, which shipped in 2015. I would expect any potential regressions to occur with the combination of a current os-brick with an older (Hammer timeframe) Ceph, as that is when this feature was introduced. the only potential regression I see in the previously mentioned timeframe would be with connecting to the older Ceph. ------------------------------------------------------------------------- https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/job-output.txt 2020-03-02 18:07:56.142233 | primary | 2020-03-02 18:07:52.442 135023 WARNING urllib3.connectionpool [-] Retrying (Retry(total=8, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))': /v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file 2020-03-02 18:07:56.142265 | primary | 2020-03-02 18:07:55.915 135023 INFO tempest.lib.common.rest_client [req-2c4e9f5f-5497-4eae-b26c-9203f8fbc85b ] Request (main): 500 PUT http://192.168.24.1:9292/v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file 14.582s 2020-03-02 18:07:56.142297 | primary | 2020-03-02 18:07:55.917 135023 CRITICAL tempest [-] Unhandled error: tempest.lib.exceptions.ServerFault: Got server fault 2020-03-02 18:07:56.142328 | primary | Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n ' 2020-03-02 18:07:56.142377 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Traceback (most recent call last): 2020-03-02 18:07:56.142411 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/bin/discover-tempest-config", line 10, in <module> 2020-03-02 18:07:56.142442 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest sys.exit(main()) 2020-03-02 18:07:56.142473 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 605, in main 2020-03-02 18:07:56.142505 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest verbose=args.verbose 2020-03-02 18:07:56.142549 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 545, in config_tempest 2020-03-02 18:07:56.142582 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image.create_tempest_images(conf) 2020-03-02 18:07:56.142612 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 113, in create_tempest_images 2020-03-02 18:07:56.142644 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image_dest=img_path) 2020-03-02 18:07:56.142675 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 161, in find_or_upload_image 2020-03-02 18:07:56.142706 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image = self._upload_image(image_name, image_dest) 2020-03-02 18:07:56.142737 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 212, in _upload_image 2020-03-02 18:07:56.142769 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self.client.store_image_file(image['id'], data) 2020-03-02 18:07:56.142801 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 151, in store_image_file 2020-03-02 18:07:56.142832 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest body=data, chunked=True) 2020-03-02 18:07:56.142863 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request 2020-03-02 18:07:56.142894 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self._error_checker(resp, resp_body) 2020-03-02 18:07:56.142926 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 864, in _error_checker 2020-03-02 18:07:56.142965 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest message=message) 2020-03-02 18:07:56.142996 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest tempest.lib.exceptions.ServerFault: Got server fault 2020-03-02 18:07:56.143049 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n ' 2020-03-02 18:07:56.143083 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest  2020-03-02 18:07:56.143113 | primary | 2020-03-02 18:07:56.143144 | primary | 2020-03-02 18:07:56.143192 | primary | STDERR: 2020-03-02 18:07:56.143225 | primary | 2020-03-02 18:07:56.143256 | primary | /usr/lib/python3.6/site-packages/config_tempest/profile.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. 2020-03-02 18:07:56.143286 | primary | return yaml.load(stream) ===================================== Actual error: is here: https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/logs/undercloud/var/log/containers/glance/api.log InvalidArgumentError('RADOS invalid argument (error calling conf_read_file)',) trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:156 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Exception while accessing to cinder volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder Traceback (most recent call last): 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data']) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder result = f(*args, **kwargs) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_cluster_name=str(cluster_name)) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__ 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder self.client, self.ioctx = self.connect() 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder conffile=self.rbd_conf) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 692, in rados.Rados.__init__ 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 743, in rados.Rados.__setup 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 826, in rados.Rados.conf_read_file 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder 2020-03-02 18:07:52.296 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to write to volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.386 40 ERROR glance.api.v2.image_data [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to upload image data due to internal error: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Caught error: [errno 22] RADOS invalid argument (error calling conf_read_file): rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi Traceback (most recent call last): 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi request, **action_args) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1390, in dispatch 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return method(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 415, in wrapped 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image-------------------------------------------------------------------------_data.py", line 299, in upload 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._restore(image_repo, image) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 164, in upload 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 501, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 448, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 204, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 319, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 548, in set_data 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 461, in _upload_to_store 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi verifier=verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 396, in add_with_multihash 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 478, in store_add_to_backend_with_multihash 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 747, in add 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f: 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return next(self.gen) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi device = conn.connect_volume(connection_info['data']) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi result = f(*args, **kwargs) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_handle = self._get_rbd_handle(connection_properties) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_cluster_name=str(cluster_name)) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.client, self.ioctx = self.connect() 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi conffile=self.rbd_conf) 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 692, in rados.Rados.__init__ 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 743, in rados.Rados.__setup 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 826, in rados.Rados.conf_read_file 2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-09-28 07:07:29 Chris MacNaughton bug added subscriber Ubuntu Stable Release Updates Team
2020-10-01 23:37:56 Seth Tanner bug added subscriber Seth Tanner
2020-10-02 09:26:02 Timo Aaltonen python-os-brick (Ubuntu Focal): status Triaged Fix Committed
2020-10-02 09:26:07 Timo Aaltonen bug added subscriber SRU Verification
2020-10-02 09:26:11 Timo Aaltonen tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal
2020-10-02 12:39:11 Corey Bryant cloud-archive/ussuri: status Triaged Fix Committed
2020-10-02 12:39:15 Corey Bryant tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal verification-ussuri-needed
2020-10-06 19:27:47 Seth Tanner tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal verification-ussuri-needed alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal verification-ussuri-done
2020-10-06 19:33:46 Seth Tanner tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-needed verification-needed-focal verification-ussuri-done alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-done-focal verification-needed verification-ussuri-done
2020-10-07 09:29:43 Chris MacNaughton tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-done-focal verification-needed verification-ussuri-done alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-done verification-done-focal verification-ussuri-done
2020-10-12 08:40:00 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2020-10-12 08:50:04 Launchpad Janitor python-os-brick (Ubuntu Focal): status Fix Committed Fix Released
2020-11-02 14:30:54 Corey Bryant cloud-archive/ussuri: status Fix Committed Fix Released
2021-06-02 09:33:13 Christian Rohmann bug added subscriber Christian Rohmann
2021-06-21 12:31:30 Aurelien Lourot charm-ceph-rbd-mirror: status New In Progress
2021-06-21 12:31:50 Aurelien Lourot charm-ceph-rbd-mirror: importance Undecided Medium
2021-06-21 19:13:41 Aurelien Lourot charm-ceph-rbd-mirror: assignee Aurelien Lourot (aurelien-lourot)
2021-07-30 10:53:02 OpenStack Infra tags alert in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-done verification-done-focal verification-ussuri-done alert in-stable-train in-stable-ussuri in-stable-victoria promotion-blocker tempest verification-done verification-done-focal verification-ussuri-done