glance image import failed for cinder backend with ceph missing cinder keyring

[Xing Zhang]

Environment:
/etc/kolla/globals.yml
enable_cinder: "yes"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"

confirmed this issue with version: master and wallaby, other branches may have the same issue.

How to reproduce:
glance-api with cinder and ceph backend, import image with glance Interoperable Image Import[1] with the parameters[2]:
all_stores: true
all_stores_must_succeed: true

the main issue is cinder backend for glance, we can test only the cinder store for glance like this:

1. Get all stores info:
# glance stores-info
stores | [{"id": "http", "read-only": "true"}, {"id": "cinder"}, {"id": "rbd", "default": "true"}]

2. Get all available import info:
# glance import-info
import-methods | {"description": "Import methods available.", "type": "array", "value": ["glance-direct", "web-download", "copy-image"]}

3. Create an empty image:
# openstack image create cirros
id | e29a7838-f4ec-445e-a1a9-79db32cbb20d

4. Import image with cinder store:
# glance image-import --import-method web-download --uri http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img --store cinder e29a7838-f4ec-445e-a1a9-79db32cbb20d

with the config in globals.yml, here is the glance-api.conf
[DEFAULT]
enabled_backends = http:http, rbd:rbd, cinder:cinder

[glance_store]
default_backend = rbd

[rbd]
rbd_store_user = glance
rbd_store_pool = images
rbd_store_chunk_size = 8

Error logs:
1. Privsep daemon failed to start
failed with Privsep daemon failed to start, full log is at [3]
glance-api.log
2023-06-24 12:39:21.087 104 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 591, in helper_main
    Daemon(channel, context).run()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 394, in run
    self._drop_privs()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs
    capabilities.drop_all_caps_except(self.caps, self.caps, [])
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
    raise OSError(errno, os.strerror(errno))
PermissionError: [Errno 1] Operation not permitted

2. cinder keyring file is missing
after changing glance-api container privileged to true, reconfigure with kolla-ansible, import the image again, got failed with FileNotFoundError, full log is at [4]
glance-api.log
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder [-] Exception while accessing to cinder volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickExcept
ion: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 75, in _check_or_get_keyring_co
ntents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder with open(keyring_path, 'r') as keyring_file:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/ceph.client.cinder.keyring'
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder During handling of the above exception, another exception occurred:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py", line 658, in _open_cinder_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data'])
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 141, in trace_logging_wrapper
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder return f(*args, **kwargs)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 244, in connect_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 123, in _get_rbd_handle
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder conf = self._create_ceph_conf(monitor_ips, monitor_ports,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 91, in _create_ceph_conf
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder keyring = cls._check_or_get_keyring_contents(keyring, cluster_name,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 82, in _check_or_get_keyring_contents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder raise exception.BrickException(msg=msg)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.820 49 ERROR glance_store._drivers.cinder [-] Failed to write to volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.944 49 WARNING glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f failed with exception An unknown exception occurred.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.946 49 INFO glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f: Could not import image file file:///var/lib/glance/staging/61931991-e93c-48ff-8791-84101525c427
2023-06-19 21:55:38.961 49 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-cinder-ImportToStore-3385c093-8d43-45cf-9724-583809ae931f' (745c1f6a-beba-4e2f-90ce-bb080f2ef7a6) transitioned into state 'FAILURE' from state 'RUNNING'

[1] https://docs.openstack.org/glance/latest/admin/interoperable-image-import.html
[2] https://docs.openstack.org/api-ref/image/v2/index.html?expanded=stage-binary-image-data-detail,import-an-image-detail#id61
[3] https://paste.openstack.org/show/820453/
[4] https://paste.openstack.org/show/820454/

[Maksim Malchuk (mmalchuk)]

please provide the versions used and how to reproduce the error.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/886906
Committed: https://opendev.org/openstack/kolla-ansible/commit/70c57d3307400759123cc9ee1542258d0ab40c0a
Submitter: "Zuul (22348)"
Branch: master

commit 70c57d3307400759123cc9ee1542258d0ab40c0a
Author: Xing Zhang <email address hidden>
Date: Sun Jun 25 11:32:24 2023 +0800

    Fix glance-api failed to start privsep daemon

    When cinder_backend_ceph is set to true, privileged is needed
    for glance-api.

    Change-Id: I92c2744d2475924ef3a2c7ae6cea7e44461de424
    Partial-Bug: #2024541

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901573

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/901701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901573
Committed: https://opendev.org/openstack/kolla-ansible/commit/2a516bde46b561fda1a9c04dc2879e6f54a536ff
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2a516bde46b561fda1a9c04dc2879e6f54a536ff
Author: Xing Zhang <email address hidden>
Date: Sun Jun 25 11:32:24 2023 +0800

    Fix glance-api failed to start privsep daemon

    When cinder_backend_ceph is set to true, privileged is needed
    for glance-api.

    Change-Id: I92c2744d2475924ef3a2c7ae6cea7e44461de424
    Partial-Bug: #2024541
    (cherry picked from commit 70c57d3307400759123cc9ee1542258d0ab40c0a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901580
Committed: https://opendev.org/openstack/kolla-ansible/commit/d3773f8ae102e379fd0d72aa4432b3a151f8aca3
Submitter: "Zuul (22348)"
Branch: stable/zed

commit d3773f8ae102e379fd0d72aa4432b3a151f8aca3
Author: Xing Zhang <email address hidden>
Date: Sun Jun 25 11:32:24 2023 +0800

    Fix glance-api failed to start privsep daemon

    When cinder_backend_ceph is set to true, privileged is needed
    for glance-api.

    Change-Id: I92c2744d2475924ef3a2c7ae6cea7e44461de424
    Partial-Bug: #2024541
    (cherry picked from commit 70c57d3307400759123cc9ee1542258d0ab40c0a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/901701
Committed: https://opendev.org/openstack/kolla-ansible/commit/64ef9451fe53a15eb8c0d6c3e739157e86465bf7
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 64ef9451fe53a15eb8c0d6c3e739157e86465bf7
Author: Xing Zhang <email address hidden>
Date: Sun Jun 25 11:32:24 2023 +0800

    Fix glance-api failed to start privsep daemon

    When cinder_backend_ceph is set to true, privileged is needed
    for glance-api.

    Change-Id: I92c2744d2475924ef3a2c7ae6cea7e44461de424
    Partial-Bug: #2024541
    (cherry picked from commit 70c57d3307400759123cc9ee1542258d0ab40c0a)