OpenStack-Ansible

When glance is configured to use cinder as a store, creating image fails in Antelope

Bug #2043503 reported by Howard Johnson on 2023-11-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack-Ansible	Fix Released	Undecided	Unassigned

Bug Description

Attempting to create an image with "openstack image create cirros-0.6.1 --file cirros-0.6.1-x86_64-disk.img" fails with "HttpException: 500: Server Error for url: http://<hidden>:9292/v2/images/51db54d6-3d6a-49e5-9045-08c281cd52b1/file, Internal Server Error". The full log is as follows:

Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.007 353973 INFO oslo.privsep.daemon [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/glance/glance-api-paste.ini', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpgy4bxosv/privsep.sock']
Nov 14 06:03:27 infra02 sudo[356149]: glance : PWD=/ ; USER=root ; COMMAND=/openstack/venvs/glance-27.2.0/bin/glance-rootwrap /etc/glance/rootwrap.conf privsep-helper --config-file /etc/glance/glance-api-paste.ini --config-file /etc/glance/glance-api.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpgy4bxosv/privsep.sock
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.359 353973 WARNING oslo.privsep.daemon [-] privsep log: /openstack/venvs/glance-27.2.0/bin/glance-rootwrap: Executable not found: privsep-helper (filter match = privsep-helper)
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.395 353973 CRITICAL oslo.privsep.daemon [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] privsep helper command exited non-zero (96)
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.396 353973 DEBUG os_brick.utils [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] <== get_connector_properties: exception (389ms) FailedToDropPrivileges('privsep helper command exited non-zero (96)') trace_logging_wrapper /openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/utils.py:188
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.396 353973 ERROR glance_store._drivers.cinder.store [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Failed to write to volume ac99e483-83d5-4028-9836-32c8d29d3b0b.: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (96)
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.479 353973 ERROR glance.api.v2.image_data [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Failed to upload image data due to internal error: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (96)
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Caught error: privsep helper command exited non-zero (96): oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (96)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi Traceback (most recent call last):
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/common/wsgi.py", line 1286, in __call__
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/common/wsgi.py", line 1329, in dispatch
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return method(*args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/common/utils.py", line 414, in wrapped
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/v2/image_data.py", line 302, in upload
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self._restore(image_repo, image)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self.force_reraise()
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi raise self.value
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/v2/image_data.py", line 162, in upload
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/notifier.py", line 497, in set_data
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self.force_reraise()
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi raise self.value
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/notifier.py", line 443, in set_data
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend,
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/quota/__init__.py", line 322, in set_data
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend,
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/location.py", line 585, in set_data
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/location.py", line 476, in _upload_to_store
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash(
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return store_add_to_backend_with_multihash(
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add(
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/driver.py", line 277, in add_adapter
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/capabilities.py", line 176, in op_checker
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 1037, in add
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self._offline_extend(client, volume, write_props)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 931, in _offline_extend
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return next(self.gen)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 713, in _open_cinder_volume
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi connector_prop = connector.get_connector_properties(
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/utils.py", line 185, in trace_logging_wrapper
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi result = f(*args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/initiator/connector.py", line 232, in get_connector_properties
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi connector.get_connector_properties(
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/initiator/connectors/iscsi.py", line 69, in get_connector_properties
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi initiator = iscsi.get_initiator()
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/initiator/connectors/iscsi.py", line 994, in get_initiator
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi lines, _err = self._execute('cat', file_path, run_as_root=True,
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/executor.py", line 55, in _execute
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi result = self.__execute(*args, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi return execute_root(*cmd, **kwargs)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi self.start()
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_privsep/priv_context.py", line 283, in start
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi channel = daemon.RootwrapClientChannel(context=self)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi File "/openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/oslo_privsep/daemon.py", line 358, in __init__
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi raise FailedToDropPrivileges(msg)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (96)
                                                 2023-11-14 06:03:27.512 353973 ERROR glance.common.wsgi
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.527 353973 DEBUG glance.api.middleware.version_negotiation [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Determining version of request: DELETE /v2/images/a33239f0-956a-4f2b-b32a-0580ff6a7ae3 Accept: */* process_request /openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/middleware/version_negotiation.py:44
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.527 353973 DEBUG glance.api.middleware.version_negotiation [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Using url versioning process_request /openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/middleware/version_negotiation.py:57
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.527 353973 DEBUG glance.api.middleware.version_negotiation [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] Matched version: v2 process_request /openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/middleware/version_negotiation.py:69
Nov 14 06:03:27 infra02 glance-wsgi-api[353973]: 2023-11-14 06:03:27.528 353973 DEBUG glance.api.middleware.version_negotiation [None req-23785320-f75c-49f6-adc1-a4aa603236de a29ba6a1679149ef860f1421e668044a c5080274158a4e6680f74984c8df33a7 - - default default] new path /v2/images/a33239f0-956a-4f2b-b32a-0580ff6a7ae3 process_request /openstack/venvs/glance-27.2.0/lib64/python3.9/site-packages/glance/api/middleware/version_negotiation.py:70

Yet privsep-helper seems to be in the system:

[root@infra02 ~]# find / -name \*privsep-helper*
/openstack/venvs/glance-27.2.0/bin/privsep-helper
/openstack/venvs/cinder-27.2.0/bin/privsep-helper
/openstack/venvs/nova-27.2.0/bin/privsep-helper
/openstack/venvs/neutron-27.2.0/bin/privsep-helper
/openstack/venvs/ceilometer-27.2.0/bin/privsep-helper

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-14: Fix proposed to openstack-ansible-os_glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/900930

Changed in openstack-ansible:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-21: Fix merged to openstack-ansible-os_glance (master)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/900930
Committed: https://opendev.org/openstack/openstack-ansible-os_glance/commit/c2428ab8da9cc3868b5ae86140a63e4a33e28eca
Submitter: "Zuul (22348)"
Branch: master

commit c2428ab8da9cc3868b5ae86140a63e4a33e28eca
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Nov 14 18:49:27 2023 +0100

Add glance_bin to rootwrap defenition

    At the moment rootwrap can not find privsep-helper binary as it's located
    in glance bin directory, which is inside the virtual environemnt.
    In order to properly use privsep we must define venv bin directory in
    allowed exec_dirs of rootwrap.

This also introduces new variable `glance_rootwrap_conf_overrides`
that allows to manage some extra overrides for rootwrap if needed.

Closes-Bug: #2043503
Change-Id: I4ee3fc33fdbeb50fc7b102bf62d6134f83c5925f

Changed in openstack-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-21: Fix proposed to openstack-ansible-os_glance (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901562

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-21: Fix proposed to openstack-ansible-os_glance (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901563

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-21: Fix proposed to openstack-ansible-os_glance (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901564

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-22: Fix merged to openstack-ansible-os_glance (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901562
Committed: https://opendev.org/openstack/openstack-ansible-os_glance/commit/a5677359abd57731d1004286e520c10b7c86effc
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit a5677359abd57731d1004286e520c10b7c86effc
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Nov 14 18:49:27 2023 +0100

Add glance_bin to rootwrap defenition

This also introduces new variable `glance_rootwrap_conf_overrides`
that allows to manage some extra overrides for rootwrap if needed.

    Closes-Bug: #2043503
    Change-Id: I4ee3fc33fdbeb50fc7b102bf62d6134f83c5925f
    (cherry picked from commit c2428ab8da9cc3868b5ae86140a63e4a33e28eca)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-23: Fix merged to openstack-ansible-os_glance (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901563
Committed: https://opendev.org/openstack/openstack-ansible-os_glance/commit/e86ce26c50f365b0a4a857ec42fe5aa444860f9b
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e86ce26c50f365b0a4a857ec42fe5aa444860f9b
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Nov 14 18:49:27 2023 +0100

Add glance_bin to rootwrap defenition

This also introduces new variable `glance_rootwrap_conf_overrides`
that allows to manage some extra overrides for rootwrap if needed.

    Closes-Bug: #2043503
    Change-Id: I4ee3fc33fdbeb50fc7b102bf62d6134f83c5925f
    (cherry picked from commit c2428ab8da9cc3868b5ae86140a63e4a33e28eca)

tags:	added: in-stable-zed
tags:	added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-23: Fix merged to openstack-ansible-os_glance (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901564
Committed: https://opendev.org/openstack/openstack-ansible-os_glance/commit/df9270951f9125c02f0840afad51ffbedafcfaba
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit df9270951f9125c02f0840afad51ffbedafcfaba
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Nov 14 18:49:27 2023 +0100

Add glance_bin to rootwrap defenition

This also introduces new variable `glance_rootwrap_conf_overrides`
that allows to manage some extra overrides for rootwrap if needed.

    Closes-Bug: #2043503
    Change-Id: I4ee3fc33fdbeb50fc7b102bf62d6134f83c5925f
    (cherry picked from commit c2428ab8da9cc3868b5ae86140a63e4a33e28eca)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-02-08: Fix included in openstack/openstack-ansible-os_glance yoga-eom

This issue was fixed in the openstack/openstack-ansible-os_glance yoga-eom release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.