| 2023-06-24 17:09:03 |
Xing Zhang |
description |
glance-api with cinder and ceph backend, import image with glance Interoperable Image Import[1] with the parameters[2]:
all_stores: true
all_stores_must_succeed: true
here is the glance-api.conf
[DEFAULT]
enabled_backends = http:http, rbd:rbd, cinder:cinder
[glance_store]
default_backend = rbd
[rbd]
rbd_store_user = glance
rbd_store_pool = images
rbd_store_chunk_size = 8
glance-api.log
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder [-] Exception while accessing to cinder volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickExcept
ion: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 75, in _check_or_get_keyring_co
ntents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder with open(keyring_path, 'r') as keyring_file:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/ceph.client.cinder.keyring'
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder During handling of the above exception, another exception occurred:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py", line 658, in _open_cinder_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data'])
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 141, in trace_logging_wrapper
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder return f(*args, **kwargs)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 244, in connect_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 123, in _get_rbd_handle
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder conf = self._create_ceph_conf(monitor_ips, monitor_ports,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 91, in _create_ceph_conf
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder keyring = cls._check_or_get_keyring_contents(keyring, cluster_name,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 82, in _check_or_get_keyring_contents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder raise exception.BrickException(msg=msg)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.820 49 ERROR glance_store._drivers.cinder [-] Failed to write to volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.944 49 WARNING glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f failed with exception An unknown exception occurred.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.946 49 INFO glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f: Could not import image file file:///var/lib/glance/staging/61931991-e93c-48ff-8791-84101525c427
2023-06-19 21:55:38.961 49 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-cinder-ImportToStore-3385c093-8d43-45cf-9724-583809ae931f' (745c1f6a-beba-4e2f-90ce-bb080f2ef7a6) transitioned into state 'FAILURE' from state 'RUNNING'
[1] https://docs.openstack.org/glance/latest/admin/interoperable-image-import.html
[2] https://docs.openstack.org/api-ref/image/v2/index.html?expanded=stage-binary-image-data-detail,import-an-image-detail#id61 |
Environment:
/etc/kolla/globals.yml
enable_cinder: "yes"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
confirmed this issue with version: master and wallaby, other branches may have the same issue.
How to reproduce:
glance-api with cinder and ceph backend, import image with glance Interoperable Image Import[1] with the parameters[2]:
all_stores: true
all_stores_must_succeed: true
the main issue is cinder backend for glance, we can test only the cinder store for glance like this:
1. Get all stores info:
# glance stores-info
stores | [{"id": "http", "read-only": "true"}, {"id": "cinder"}, {"id": "rbd", "default": "true"}]
2. Get all available import info:
# glance import-info
import-methods | {"description": "Import methods available.", "type": "array", "value": ["glance-direct", "web-download", "copy-image"]}
3. Create an empty image:
# openstack image create cirros
id | e29a7838-f4ec-445e-a1a9-79db32cbb20d
4. Import image with cinder store:
# glance image-import --import-method web-download --uri http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img --store cinder e29a7838-f4ec-445e-a1a9-79db32cbb20d
with the config in globals.yml, here is the glance-api.conf
[DEFAULT]
enabled_backends = http:http, rbd:rbd, cinder:cinder
[glance_store]
default_backend = rbd
[rbd]
rbd_store_user = glance
rbd_store_pool = images
rbd_store_chunk_size = 8
Error logs:
1. Privsep daemon failed to start
failed with Privsep daemon failed to start, full log is at [3]
glance-api.log
2023-06-24 12:39:21.087 104 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted
Traceback (most recent call last):
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 591, in helper_main
Daemon(channel, context).run()
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 394, in run
self._drop_privs()
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 430, in _drop_privs
capabilities.drop_all_caps_except(self.caps, self.caps, [])
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
raise OSError(errno, os.strerror(errno))
PermissionError: [Errno 1] Operation not permitted
2. cinder keyring file is missing
after changing glance-api container privileged to true, reconfigure with kolla-ansible, import the image again, got failed with FileNotFoundError, full log is at [4]
glance-api.log
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder [-] Exception while accessing to cinder volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickExcept
ion: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 75, in _check_or_get_keyring_co
ntents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder with open(keyring_path, 'r') as keyring_file:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder FileNotFoundError: [Errno 2] No such file or directory: '/etc/ceph/ceph.client.cinder.keyring'
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder During handling of the above exception, another exception occurred:
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py", line 658, in _open_cinder_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder device = conn.connect_volume(connection_info['data'])
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 141, in trace_logging_wrapper
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder return f(*args, **kwargs)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 244, in connect_volume
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder rbd_handle = self._get_rbd_handle(connection_properties)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 123, in _get_rbd_handle
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder conf = self._create_ceph_conf(monitor_ips, monitor_ports,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 91, in _create_ceph_conf
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder keyring = cls._check_or_get_keyring_contents(keyring, cluster_name,
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/rbd.py", line 82, in _check_or_get_keyring_contents
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder raise exception.BrickException(msg=msg)
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.535 49 ERROR glance_store._drivers.cinder
2023-06-19 21:55:38.820 49 ERROR glance_store._drivers.cinder [-] Failed to write to volume 1ffc6374-4bb3-4ff2-bdc8-483755ff9931.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.944 49 WARNING glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f failed with exception An unknown exception occurred.: os_brick.exception.BrickException: An unknown exception occurred.
2023-06-19 21:55:38.946 49 INFO glance.common.scripts.image_import.main [-] Task 3385c093-8d43-45cf-9724-583809ae931f: Could not import image file file:///var/lib/glance/staging/61931991-e93c-48ff-8791-84101525c427
2023-06-19 21:55:38.961 49 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-cinder-ImportToStore-3385c093-8d43-45cf-9724-583809ae931f' (745c1f6a-beba-4e2f-90ce-bb080f2ef7a6) transitioned into state 'FAILURE' from state 'RUNNING'
[1] https://docs.openstack.org/glance/latest/admin/interoperable-image-import.html
[2] https://docs.openstack.org/api-ref/image/v2/index.html?expanded=stage-binary-image-data-detail,import-an-image-detail#id61
[3] https://paste.openstack.org/show/820453/
[4] https://paste.openstack.org/show/820454/ |
|
