[SRU] tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument

[wes hayutin]

[Impact]

Os-brick is unusable with Ceph Octopus because of a tightening of config formatting requirements in Ceph.

[Test Case]

Validate cinder-backup with Ceph on Ubuntu Focal, in addition to testing with Ceph on Xenial (OpenStack Mitaka) with Cinder (and cinder-backup) on Focal (OpenStack Ussuri)

[Regression Potential]

The risk should be fairly low as the new confiig line being generated is supported since Ceph Hammer, which shipped in 2015. I would expect any potential regressions to occur with the combination of a current os-brick with an older (Hammer timeframe) Ceph, as that is when this feature was introduced.

the only potential regression I see in the previously mentioned timeframe would be with connecting to the older Ceph.

-------------------------------------------------------------------------

https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/job-output.txt

2020-03-02 18:07:56.142233 | primary | 2020-03-02 18:07:52.442 135023 WARNING urllib3.connectionpool [-] Retrying (Retry(total=8, connect=None, read=None, redirect=5, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))': /v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file
2020-03-02 18:07:56.142265 | primary | 2020-03-02 18:07:55.915 135023 INFO tempest.lib.common.rest_client [req-2c4e9f5f-5497-4eae-b26c-9203f8fbc85b ] Request (main): 500 PUT http://192.168.24.1:9292/v2/images/22582e2f-9ce2-43a6-90e2-fee0ddd283e3/file 14.582s
2020-03-02 18:07:56.142297 | primary | 2020-03-02 18:07:55.917 135023 CRITICAL tempest [-] Unhandled error: tempest.lib.exceptions.ServerFault: Got server fault
2020-03-02 18:07:56.142328 | primary | Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n '
2020-03-02 18:07:56.142377 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Traceback (most recent call last):
2020-03-02 18:07:56.142411 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/bin/discover-tempest-config", line 10, in <module>
2020-03-02 18:07:56.142442 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest sys.exit(main())
2020-03-02 18:07:56.142473 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 605, in main
2020-03-02 18:07:56.142505 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest verbose=args.verbose
2020-03-02 18:07:56.142549 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/main.py", line 545, in config_tempest
2020-03-02 18:07:56.142582 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image.create_tempest_images(conf)
2020-03-02 18:07:56.142612 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 113, in create_tempest_images
2020-03-02 18:07:56.142644 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image_dest=img_path)
2020-03-02 18:07:56.142675 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 161, in find_or_upload_image
2020-03-02 18:07:56.142706 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest image = self._upload_image(image_name, image_dest)
2020-03-02 18:07:56.142737 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/config_tempest/services/image.py", line 212, in _upload_image
2020-03-02 18:07:56.142769 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self.client.store_image_file(image['id'], data)
2020-03-02 18:07:56.142801 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 151, in store_image_file
2020-03-02 18:07:56.142832 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest body=data, chunked=True)
2020-03-02 18:07:56.142863 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
2020-03-02 18:07:56.142894 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest self._error_checker(resp, resp_body)
2020-03-02 18:07:56.142926 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 864, in _error_checker
2020-03-02 18:07:56.142965 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest message=message)
2020-03-02 18:07:56.142996 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest tempest.lib.exceptions.ServerFault: Got server fault
2020-03-02 18:07:56.143049 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest Details: b'500 Internal Server Error\n\nThe server has either erred or is incapable of performing the requested operation.\n\n '
2020-03-02 18:07:56.143083 | primary | 2020-03-02 18:07:55.917 135023 ERROR tempest 
2020-03-02 18:07:56.143113 | primary |
2020-03-02 18:07:56.143144 | primary |
2020-03-02 18:07:56.143192 | primary | STDERR:
2020-03-02 18:07:56.143225 | primary |
2020-03-02 18:07:56.143256 | primary | /usr/lib/python3.6/site-packages/config_tempest/profile.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
2020-03-02 18:07:56.143286 | primary | return yaml.load(stream)

=====================================

Actual error: is here:

https://12b5c9445590716cbe81-2eb50734132c0e56282483bcdf57bf8a.ssl.cf2.rackcdn.com/710810/3/check/tripleo-ci-centos-8-scenario001-standalone/8a6d4a7/logs/undercloud/var/log/containers/glance/api.log

InvalidArgumentError('RADOS invalid argument (error calling conf_read_file)',) trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:156
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Exception while accessing to cinder volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data'])
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder result = f(*args, **kwargs)
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties)
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rbd_cluster_name=str(cluster_name))
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder self.client, self.ioctx = self.connect()
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder conffile=self.rbd_conf)
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 692, in rados.Rados.__init__
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 743, in rados.Rados.__setup
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder File "rados.pyx", line 826, in rados.Rados.conf_read_file
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-03-02 18:07:51.999 40 ERROR glance_store._drivers.cinder
2020-03-02 18:07:52.296 40 ERROR glance_store._drivers.cinder [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to write to volume 17a385a3-9331-4513-9a81-37a99200cefd.: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-03-02 18:07:52.386 40 ERROR glance.api.v2.image_data [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Failed to upload image data due to internal error: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi [req-5076b6b5-c8d6-4743-83dc-245492610376 b9f74f3fb12843818bc784bf2f81fca2 966194a384e641598d107cdf193a9a39 - default default] Caught error: [errno 22] RADOS invalid argument (error calling conf_read_file): rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi Traceback (most recent call last):
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1347, in __call__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi request, **action_args)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1390, in dispatch
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return method(*args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 415, in wrapped
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image-------------------------------------------------------------------------_data.py", line 299, in upload
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._restore(image_repo, image)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise()
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 164, in upload
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 501, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.force_reraise()
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi raise value
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 448, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 204, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 319, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi set_active=set_active)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 548, in set_data
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 461, in _upload_to_store
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi verifier=verifier)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 396, in add_with_multihash
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 478, in store_add_to_backend_with_multihash
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 747, in add
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi return next(self.gen)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py", line 554, in _open_cinder_volume
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi device = conn.connect_volume(connection_info['data'])
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi result = f(*args, **kwargs)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 203, in connect_volume
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_handle = self._get_rbd_handle(connection_properties)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rbd_cluster_name=str(cluster_name))
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi self.client, self.ioctx = self.connect()
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxrbd.py", line 73, in connect
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi conffile=self.rbd_conf)
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 692, in rados.Rados.__init__
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 743, in rados.Rados.__setup
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi File "rados.pyx", line 826, in rados.Rados.conf_read_file
2020-03-02 18:07:52.438 40 ERROR glance.common.wsgi rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)

[wes hayutin (weshayutin)]

 default default] Failed to upload image data due to internal error: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)

[yatin (yatinkarel)]

So i checked this with Abhishek Kekane today and we found that issue is related to ceph client(octopus) and os-brick. The issue happened when cinder(backend is rbd) is used as glance backend and ceph octopus client is used.

os-brick creates a temp configuration file[1] and uses it to connect to ceph cluster. With ceph octopus this do not work without [global] section defined for mon_host.

[global] <<- without this in temp configuration, ceph octopus client do not work, nautils client works fine
mon_host = <ip>:3300

octopus clients are coming as ceph master repo is used in kolla currently [2], a patch is proposed to use nautilus repo from Storage SIG https://review.opendev.org/#/c/707338/ in kolla. So atleast TripleO jobs shouldn't see this issue after that patch is merged and new container images are build from it.

And fixing(add [global] section while create config file) in os-brick could also be done as that should work with both nautilus and octopus ceph. Me don't know if this scenario is tested in glance/cinder in non TripleO job and if ceph octopus or ceph nautilus is used there. I think this is not being tested upstream otherwise the issue would have caught already or may be it's handled in different way.

[1] https://github.com/openstack/os-brick/blob/a6dff156d70cd79e87c71bea53aa634311f22459/os_brick/initiator/connectors/rbd.py#L98-L99
[2] https://github.com/openstack/kolla/blob/master/docker/base/ceph_master.repo

[John Fulton (jfulton-org)]

So the cinder or glance containers have octopus clients and moving them to nautilus should address the bug. FWIW I'm hoping that victoria can be tested with octopus.

[Giulio Fidente (gfidente)]

Looks like https://review.opendev.org/#/c/707338/ merged

[yatin (yatinkarel)]

<< Looks like https://review.opendev.org/#/c/707338/ merged
Yes kolla patch merged, and new containers are build and scenario001 passing in promotion jobs:-
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ecb22ff/logs/undercloud/var/log/tempest/stestr_results.html.gz

Once new containers are promoted to docker hub, scenario001 should pass in check jobs too.

[Corey Bryant (corey.bryant)]

Should this be targeted at the upstream os-brick project to fix this everywhere?

[Alex Kavanagh (ajkavanagh)]

Note I've pushed this review as a fix for this:

https://review.opendev.org/#/c/730376/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717068
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-rbd-mirror/commit/?id=5199c767137fe4fa2abcd619221facad40c6defe
Submitter: Zuul
Branch: master

commit 5199c767137fe4fa2abcd619221facad40c6defe
Author: Alex Kavanagh <email address hidden>
Date: Thu Apr 2 16:59:07 2020 +0100

    Add focal and ussuri bundles to the charm

    This patch updates the bundles to include up to focal-ussuri.
    The focal-ussuri bundle is in the dev bundles as it can't pass at the
    moment due to LP: #1865754.

    The bionic-ussuri bundle is in the dev bundles (i.e. not gate) as it
    fails due to LP: #1892201

    Also deal with the related bug where cinder-ceph requires the relation
    with a nova-compute unit.

    Related-Bug: #1881246
    Related-Bug: #1865754
    Related-Bug: #1892201
    Change-Id: I0a6f1de82ecc601509822277d657485e08dc893d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730376
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=3e831480c29c07128f36ae67c44c25177b3d9f36
Submitter: Zuul
Branch: master

commit 3e831480c29c07128f36ae67c44c25177b3d9f36
Author: Alex Kavanagh <email address hidden>
Date: Fri May 22 17:30:45 2020 +0100

    Fix for rbd connector to work with ceph octopus

    Found whilst testing cinder-backup with ceph on Ubuntu focal and
    OpenStack Ussuri. Focal installs Ceph Octopus, and, as the linked bug
    shows, Octopus has slightly changed its requirements to need a
    [global] ini heading at the beginning of the config file that is
    generated within the RBDConnector() class.

    Includes code to resolve which version of ceph is installed as a client
    to only change octopus and later.

    It also fixes cinder Ceph backup driver.

    Includes release note for the change.

    Co-Authored-By: Ivan Kolodyazhny <email address hidden>
    Change-Id: Ie8011fc83bc41edd2768ade6d42d737b4598ca2d
    Closes-Bug: #1865754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/749697

[Brian Rosmaita (brian-rosmaita)]

Change-Id Ie8011fc83bc41edd2768ade6d42d737b4598ca2d added a pretty heavy-duty binary dependency to os-brick. In the meantime, we have verified that use of the '[global]' section of the config file goes back at least to hammer, which would be in the openstack liberty time-frame:

https://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/

So I think we need to revert commit 3e831480c29c07128f36ae67c44c25177b3d9f36 from master and replace it with a change that unconditionally adds the '[global]' section to the generated config file, and then backport both changes to stable/victoria.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/750444

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/750452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750444
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=c54c41a1fd69b5c191b12a8da0eb12f2ee561f10
Submitter: Zuul
Branch: master

commit c54c41a1fd69b5c191b12a8da0eb12f2ee561f10
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:16:52 2020 -0400

    Revert "Fix for rbd connector to work with ceph octopus"

    This reverts commit 3e831480c29c07128f36ae67c44c25177b3d9f36.

    This change introduced a heavyweight bindep to os-brick. We will
    replace it with a more lightweight fix.

    Change-Id: I9eff1d0d8bab7aaf060d2c009bed2be65d212c04
    Partial-bug: #1865754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/750649

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/750655

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750452
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=c6ad4d864c6088b6686919dd0229c926bce7073a
Submitter: Zuul
Branch: master

commit c6ad4d864c6088b6686919dd0229c926bce7073a
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:44:25 2020 -0400

    New fix for rbd connector to work with ceph octopus

    Found while testing cinder-backup with ceph on Ubuntu Focal, which
    installs ceph Octopus. Octopus apparently enforces the requirement
    that the config file contain a '[global]' section for general
    requirements. The '[global]' section goes back at least to ceph
    Hammer [0], so we will simply add it to the temporary ceph config
    file that os-brick generates in the RBDConnector class.

    [0] https://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/

    Co-authored-by: Alex Kavanagh <email address hidden>
    Co-authored-by: Ivan Kolodyazhny <email address hidden>

    Change-Id: I86eb31535d990291945de5d9846b1a03157ec2cf
    Closes-bug: #1865754

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750649
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=8d297ddad3b0e40c6beede8e1460f7105da0c273
Submitter: Zuul
Branch: stable/victoria

commit 8d297ddad3b0e40c6beede8e1460f7105da0c273
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:16:52 2020 -0400

    Revert "Fix for rbd connector to work with ceph octopus"

    This reverts commit 3e831480c29c07128f36ae67c44c25177b3d9f36.

    This change introduced a heavyweight bindep to os-brick. We will
    replace it with a more lightweight fix.

    Change-Id: I9eff1d0d8bab7aaf060d2c009bed2be65d212c04
    Partial-bug: #1865754
    (cherry picked from commit c54c41a1fd69b5c191b12a8da0eb12f2ee561f10)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750655
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=474583b4f8469f7957d6ffbf62829b516189aeae
Submitter: Zuul
Branch: stable/victoria

commit 474583b4f8469f7957d6ffbf62829b516189aeae
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:44:25 2020 -0400

    New fix for rbd connector to work with ceph octopus

    Found while testing cinder-backup with ceph on Ubuntu Focal, which
    installs ceph Octopus. Octopus apparently enforces the requirement
    that the config file contain a '[global]' section for general
    requirements. The '[global]' section goes back at least to ceph
    Hammer [0], so we will simply add it to the temporary ceph config
    file that os-brick generates in the RBDConnector class.

    [0] https://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/

    Co-authored-by: Alex Kavanagh <email address hidden>
    Co-authored-by: Ivan Kolodyazhny <email address hidden>

    Change-Id: I86eb31535d990291945de5d9846b1a03157ec2cf
    Closes-bug: #1865754
    (cherry picked from commit c6ad4d864c6088b6686919dd0229c926bce7073a)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Rosmaita (<email address hidden>) on branch: stable/ussuri
Review: https://review.opendev.org/749697
Reason: Backport https://review.opendev.org/#/c/750655/ instead.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/751246

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751246
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=91c73a433a2b3675e661f1fe007b58c2bfaff757
Submitter: Zuul
Branch: stable/ussuri

commit 91c73a433a2b3675e661f1fe007b58c2bfaff757
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:44:25 2020 -0400

    New fix for rbd connector to work with ceph octopus

    Found while testing cinder-backup with ceph on Ubuntu Focal, which
    installs ceph Octopus. Octopus apparently enforces the requirement
    that the config file contain a '[global]' section for general
    requirements. The '[global]' section goes back at least to ceph
    Hammer [0], so we will simply add it to the temporary ceph config
    file that os-brick generates in the RBDConnector class.

    [0] https://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/

    Co-authored-by: Alex Kavanagh <email address hidden>
    Co-authored-by: Ivan Kolodyazhny <email address hidden>

    Change-Id: I86eb31535d990291945de5d9846b1a03157ec2cf
    Closes-bug: #1865754
    (cherry picked from commit c6ad4d864c6088b6686919dd0229c926bce7073a)
    (cherry picked from commit 474583b4f8469f7957d6ffbf62829b516189aeae)

[Timo Aaltonen (tjaalton)]

Hello wes, or anyone else affected,

Accepted python-os-brick into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-brick/3.0.1-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello wes, or anyone else affected,

Accepted python-os-brick into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seth Tanner (sjtanner)]

Corey,
I'm running focal/usrruri/octopus
There may be something that I'm missing here.
sudo add-apt-repository cloud-archive:ussuri-proposed
 Ubuntu Cloud Archive for OpenStack Ussuri [proposed]
 More info: https://wiki.ubuntu.com/OpenStack/CloudArchive
Press [ENTER] to continue or Ctrl-c to cancel adding it.

cloud-archive for Ussuri-proposed only supported on bionic

from reading through the docs on testing proposed, it appears the way to do this on focal would be
cat <<EOF | sudo tee -a /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive

deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
cat <<EOF | sudo tee -a /etc/apt/preferences.d/proposed-updates
# Configure apt to allow selective installs of packages from proposed
Package: *
Pin: release a=$(lsb_release -cs)-proposed
Pin-Priority: 400
EOF
  sudo apt-get update
# verify that proposed updated will only be installed when specifically requested
  sudo apt-get upgrade -s
  sudo apt-get install os-brick-common/focal-proposed

which appears that it would uninstall cinder

Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '3.0.1-0ubuntu1.3' (Ubuntu:20.04/focal-proposed [all]) for 'os-brick-common'
The following packages were automatically installed and are no longer required:
  alembic docutils-common libblas3 libconfig-general-perl libfreetype6 libgfortran5 libimagequant0 libjbig0
  libjpeg-turbo8 libjpeg8 libjs-sphinxdoc libjs-underscore liblapack3 liblcms2-2 libpaper-utils libpaper1 libpq5
  libtiff5 libwebp6 libwebpdemux2 libwebpmux3 pycadf-common python-babel-localedata python-pastedeploy-tpl
  python3-alembic python3-amqp python3-anyjson python3-automaton python3-babel python3-barbicanclient python3-bcrypt
  python3-cachetools python3-castellan python3-cffi python3-cliff python3-cmd2 python3-cursive python3-ddt
  python3-debtcollector python3-decorator python3-defusedxml python3-docutils python3-dogpile.cache python3-eventlet
  python3-extras python3-fasteners python3-fixtures python3-futurist python3-glanceclient python3-greenlet
  python3-iso8601 python3-kazoo python3-keystoneauth1 python3-keystoneclient python3-keystonemiddleware
  python3-kombu python3-mako python3-migrate python3-mimeparse python3-monotonic python3-msgpack python3-networkx
  python3-novaclient python3-numpy python3-olefile python3-os-service-types python3-os-win python3-oslo.cache
  python3-oslo.concurrency python3-oslo.config python3-oslo.context python3-oslo.db python3-oslo.i18n
  python3-oslo.log python3-oslo.messaging python3-oslo.middleware python3-oslo.policy python3-oslo.privsep
  python3-oslo.reports python3-oslo.rootwrap python3-oslo.serialization python3-oslo.service
  python3-oslo.upgradecheck python3-oslo.utils python3-oslo.versionedobjects python3-oslo.vmware python3-osprofiler
  python3-paramiko python3-paste python3-pastedeploy python3-pastescript python3-pil python3-ply python3-psycopg2
  python3-pycadf python3-pycparser python3-pygments python3-pyinotify python3-pymemcache python3-pymysql
  python3-pyparsing python3-pyperclip python3-redis pyt...

[Seth Tanner (sjtanner)]

Corey,
My apologies for the previous comment please disregard, I forgot that I needed to run upgrade rather than install.

after running
sudo apt-get upgrade os-brick-common/focal-proposed python3-os-brick/focal-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '3.0.1-0ubuntu1.3' (Ubuntu:20.04/focal-proposed [all]) for 'os-brick-common'
Selected version '3.0.1-0ubuntu1.3' (Ubuntu:20.04/focal-proposed [all]) for 'python3-os-brick'
Calculating upgrade... Done
The following packages will be upgraded:
  os-brick-common python3-os-brick
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Inst python3-os-brick [3.0.1-0ubuntu1.2] (3.0.1-0ubuntu1.3 Ubuntu:20.04/focal-proposed [all]) []
Inst os-brick-common [3.0.1-0ubuntu1.2] (3.0.1-0ubuntu1.3 Ubuntu:20.04/focal-proposed [all])
Conf python3-os-brick (3.0.1-0ubuntu1.3 Ubuntu:20.04/focal-proposed [all])
Conf os-brick-common (3.0.1-0ubuntu1.3 Ubuntu:20.04/focal-proposed [all])

and restarting cinder services
I am able to perform cinder backups and restore cinder backups, but it looks like there may be another issue with creating a snapshot and then creating a backup from the snapshot and then restoring that backup.

As a side note spawning a new instance from a snapshot, which creates a new volume works just fine.

Here are the logs from a failed restore, when restoring a backup that was created from a volume snapshot

2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager [req-ff639558-c9ae-4dd3-923d-ec158b08f3dc 78af964f6db708aebd4103b6ee7fbdfe2d30f6e775a0a007ee04cd4633ccb87d aea00f20ec77483d963d2e8f54a1132a - e5bf6979cc0e42fa86ce5be4f12241f1 e5bf6979cc0e42fa86ce5be4f12241f1] Restoring backup 9574fcbf-fe7c-41f9-8d28-dc46e3643aae to volume a0fd2623-9ab0-402e-a9da-d7bcc4bd3247 failed.: rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'volume-a97b9aaa-bcce-4ac2-ab84-60a6f07f4be2.backup.base' at snapshot None)
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager Traceback (most recent call last):
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 657, in _run_restore
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager backup_service.restore(backup, volume.id,
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 1264, in restore
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager self._restore_volume(backup, target_volume, volume_file)
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 1230, in _restore_volume
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager self._full_restore(backup, volume_file, volume.name,
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 1023, in _full_restore
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager src_rbd = eventlet.tpool.Proxy(self.rbd.Image(client.ioctx,
2020-10-06 18:38:17.002 951 ERROR cinder.backup.manager File "rbd.p...

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-os-brick has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-os-brick - 3.0.1-0ubuntu1.3

---------------
python-os-brick (3.0.1-0ubuntu1.3) focal; urgency=medium

  * d/p/lp1865754-fix-rbd-connector-for-octopus.patch: Fix RBD connector to
    work with Ceph Octopus (LP: #1865754).

 -- Chris MacNaughton <email address hidden> Wed, 23 Sep 2020 15:17:56 +0000

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-os-brick - 3.0.1-0ubuntu1.3~cloud0
---------------

 python-os-brick (3.0.1-0ubuntu1.3~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-os-brick (3.0.1-0ubuntu1.3) focal; urgency=medium
 .
   * d/p/lp1865754-fix-rbd-connector-for-octopus.patch: Fix RBD connector to
     work with Ceph Octopus (LP: #1865754).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 4.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 3.0.4 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/os-brick/+/794198

[Christian Rohmann (christian-rohmann)]

This issue also affects OpenStack Train - see my proposed backport at: https://bugs.launchpad.net/tripleo/+bug/1865754/comments/33

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/797287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/794198
Committed: https://opendev.org/openstack/os-brick/commit/57409c28b52ee2bff90679b4c93059bf5b5ba2b6
Submitter: "Zuul (22348)"
Branch: stable/train

commit 57409c28b52ee2bff90679b4c93059bf5b5ba2b6
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 8 11:44:25 2020 -0400

    New fix for rbd connector to work with ceph octopus

    Found while testing cinder-backup with ceph on Ubuntu Focal, which
    installs ceph Octopus. Octopus apparently enforces the requirement
    that the config file contain a '[global]' section for general
    requirements. The '[global]' section goes back at least to ceph
    Hammer [0], so we will simply add it to the temporary ceph config
    file that os-brick generates in the RBDConnector class.

    [0] https://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/

    Co-authored-by: Alex Kavanagh <email address hidden>
    Co-authored-by: Ivan Kolodyazhny <email address hidden>

    Change-Id: I86eb31535d990291945de5d9846b1a03157ec2cf
    Closes-bug: #1865754
    (cherry picked from commit c6ad4d864c6088b6686919dd0229c926bce7073a)
    (cherry picked from commit 474583b4f8469f7957d6ffbf62829b516189aeae)
    (cherry picked from commit 91c73a433a2b3675e661f1fe007b58c2bfaff757)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Aurelien Lourot <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/797287
Reason: Superseded by https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/816006

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick train-eol release.