kolla-ansible

Glance image-import direct not working

Bug #2048525 reported by Michal Arbet on 2024-01-08

This bug affects 2 people

	Status	Importance	Assigned to
kolla-ansible	Fix Released	High	Michal Arbet
Antelope	Fix Released	Undecided	Michal Arbet
Bobcat	Fix Released	Undecided	Michal Arbet
Caracal	Fix Released	High	Michal Arbet
Yoga	Fix Released	Undecided	Michal Arbet
Zed	Fix Released	Undecided	Michal Arbet

Bug Description

Hi,

By default glance has enabled_import_methods = [glance-direct,web-download,copy-image] enabled_import_methods and kolla-ansible just leave it in default

So, this should work if it is configured rigt ? :) ]

Problem is that glances are behind the proxy and public_endpoint is reported by each of them, so there is no way how to know which glance is currently importing the image (as we don't have shared storage between them BUT we of course don't need it ...as glance from some version can PROXY requests to the right one based on URL which glance return ... it's needed to just properly configure ...check below)

Sooo, check below current kolla-ansible configuration :

command :

glance image-create-via-import --container-format bare --disk-format qcow2 --name 'Ubuntu Test' --file jammy-server-cloudimg-amd64.img --import-method glance-direct

2024-01-08 11:32:31.252 51 INFO eventlet.wsgi.server [-] (51) wsgi starting up on http://192.168.205.12:9292
2024-01-08 11:32:56.823 51 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:32:56] "GET / HTTP/1.1" 300 1776 0.007984
2024-01-08 11:32:58.180 51 INFO eventlet.wsgi.server [-] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:32:58] "GET / HTTP/1.1" 300 1791 0.002321
2024-01-08 11:33:26.951 50 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:33:26] "GET / HTTP/1.1" 300 1776 0.009683
2024-01-08 11:33:43.118 51 INFO eventlet.wsgi.server [None req-e91026a9-5d50-45a1-8032-86f82f85a940 a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:33:43] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.789190
2024-01-08 11:33:43.999 51 INFO eventlet.wsgi.server [None req-38817699-b5d3-4c60-add0-54df6952b4c0 a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:33:43] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.124963
2024-01-08 11:33:44.338 51 INFO eventlet.wsgi.server [None req-98722a4c-6a05-48c5-9b69-78c20072194b a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:33:44] "POST /v2/images HTTP/1.1" 201 987 0.290914
2024-01-08 11:33:57.071 48 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:33:57] "GET / HTTP/1.1" 300 1776 0.008668
2024-01-08 11:34:27.216 47 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:34:27] "GET / HTTP/1.1" 300 1776 0.010865
2024-01-08 11:34:46.628 50 INFO eventlet.wsgi.server [-] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:34:46] "GET /versions HTTP/1.1" 200 1777 0.004108
2024-01-08 11:34:48.336 50 INFO eventlet.wsgi.server [None req-1a8e6d27-0717-47ff-9802-28e931949c0c a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:34:48] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.091328
2024-01-08 11:34:57.352 50 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:34:57] "GET / HTTP/1.1" 300 1776 0.002596
2024-01-08 11:34:58.821 47 INFO eventlet.wsgi.server [None req-24ec839e-3e0e-4285-99bf-0273383b551b a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:34:58] "POST /v2/images/04ef27cc-44be-4944-ba64-e379f292adee/import HTTP/1.1" 202 187 0.640125
2024-01-08 11:34:58.833 47 INFO glance.domain [-] Task [99ffa0d2-347a-490b-975a-6996c775e865] status changing from pending to processing
2024-01-08 11:34:59.127 47 INFO glance.common.scripts.image_import.main [-] Task 99ffa0d2-347a-490b-975a-6996c775e865: Got image data uri file:///var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee to be imported
2024-01-08 11:34:59.128 47 WARNING glance.common.scripts.image_import.main [-] Task 99ffa0d2-347a-490b-975a-6996c775e865 failed with exception [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee': FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee'
2024-01-08 11:34:59.129 47 INFO glance.common.scripts.image_import.main [-] Task 99ffa0d2-347a-490b-975a-6996c775e865: Could not import image file file:///var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee
2024-01-08 11:34:59.138 47 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-rbd-ImportToStore-99ffa0d2-347a-490b-975a-6996c775e865' (567e276e-d601-4148-8304-55be1b86157c) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
  Flow 'api_image_import-rbd'
  |__Atom 'api_image_import-ConfigureStaging-99ffa0d2-347a-490b-975a-6996c775e865' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': 'file:///var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee'}
     |__Atom 'api_image_import-ImageLock-99ffa0d2-347a-490b-975a-6996c775e865' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': None}
        |__Atom 'api_image_import_retry' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': [(None, {})]}
           |__Flow 'api_image_import': FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee'
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor Traceback (most recent call last):
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor result = task.execute(**arguments)
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/async_/flows/api_image_import.py", line 640, in execute
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor self._execute(action, file_path)
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/async_/flows/api_image_import.py", line 649, in _execute
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor action.set_image_data(file_path or self.uri,
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/async_/flows/api_image_import.py", line 342, in set_image_data
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor return image_import.set_image_data(self._image, uri, task_id,
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/common/scripts/image_import/main.py", line 153, in set_image_data
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor with excutils.save_and_reraise_exception():
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor self.force_reraise()
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor raise self.value
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/common/scripts/image_import/main.py", line 145, in set_image_data
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor data_iter = script_utils.get_image_data_iter(uri)
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor File "/var/lib/kolla/venv/lib/python3.11/site-packages/glance/common/scripts/utils.py", line 140, in get_image_data_iter
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor return open(uri, "rb")
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor ^^^^^^^^^^^^^^^
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee'
2024-01-08 11:34:59.138 47 ERROR glance.async_.taskflow_executor
2024-01-08 11:34:59.271 47 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-rbd-ImportToStore-99ffa0d2-347a-490b-975a-6996c775e865' (567e276e-d601-4148-8304-55be1b86157c) transitioned into state 'REVERTED' from state 'REVERTING'
2024-01-08 11:34:59.282 47 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-ConfigureStaging-99ffa0d2-347a-490b-975a-6996c775e865' (0e1e94f0-32e0-4c72-823c-e9822635789e) transitioned into state 'REVERTED' from state 'REVERTING'
2024-01-08 11:34:59.329 47 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-ImageLock-99ffa0d2-347a-490b-975a-6996c775e865' (f90825fa-e3d3-4edd-bc86-c2dc27adb7ed) transitioned into state 'REVERTED' from state 'REVERTING'
2024-01-08 11:34:59.339 47 WARNING glance.async_.taskflow_executor [-] Flow 'api_image_import' (e7e8d96d-d8cb-4326-88b2-7b92db06eb67) transitioned into state 'REVERTED' from state 'RUNNING'
2024-01-08 11:34:59.340 47 ERROR glance.async_.taskflow_executor [-] Failed to execute task 99ffa0d2-347a-490b-975a-6996c775e865: [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee': FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/glance/staging/04ef27cc-44be-4944-ba64-e379f292adee'
2024-01-08 11:34:59.340 47 INFO glance.domain [-] Task [99ffa0d2-347a-490b-975a-6996c775e865] status changing from processing to failure
2024-01-08 11:35:24.524 48 INFO eventlet.wsgi.server [-] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:35:24] "GET / HTTP/1.1" 300 1791 0.002943
2024-01-08 11:35:27.498 48 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:35:27] "GET / HTTP/1.1" 300 1776 0.002252
2024-01-08 11:35:57.627 48 INFO eventlet.wsgi.server [-] 192.168.205.12 - - [08/Jan/2024 11:35:57] "GET / HTTP/1.1" 300 1776 0.002412

And it will be importing forewer.

FIXed by set worker_self_reference_url to the correct value, so proxying started to work ...

/var/log/kolla/glance/glance-api.log:2024-01-08 11:43:16.527 50 INFO eventlet.wsgi.server [None req-0ac27f5c-e945-4bee-9f2e-f6ba5c1992e0 a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.5,192.168.205.11 - - [08/Jan/2024 11:43:16] "PUT /v2/images/af3e440e-19ae-4fe7-adca-a9a645ed1ee0/stage HTTP/1.1" 204 189 10.097863
/var/log/kolla/glance/glance-api.log:2024-01-08 11:43:18.281 48 INFO eventlet.wsgi.server [None req-fc5074cc-571d-4555-a6f3-296712301b2a a3a6c635af6d42b19e222dbd89b57079 59b04877e44a4d72aaf9b1683a26c1e3 - - default default] 192.168.205.11 - - [08/Jan/2024 11:43:18] "POST /v2/images/af3e440e-19ae-4fe7-adca-a9a645ed1ee0/import HTTP/1.1" 202 211 1.527723
/var/log/kolla/glance/glance-api.log:2024-01-08 11:43:18.607 48 INFO glance.common.scripts.image_import.main [-] Task a5039004-33d1-4f36-b888-0d6a274c1e8c: Got image data uri file:///var/lib/glance/staging/af3e440e-19ae-4fe7-adca-a9a645ed1ee0 to be imported

Thanks :)

OpenStack Infra (hudson-openstack) on 2024-01-08

Changed in kolla-ansible:
status:	New → In Progress

Michal Arbet (michalarbet) on 2024-01-08

Changed in kolla-ansible:
assignee:	nobody → Michal Arbet (michalarbet)
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-08: Fix proposed to kolla-ansible (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904850

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-08: Fix proposed to kolla-ansible (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904851

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-08: Fix proposed to kolla-ansible (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904852

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-08: Fix proposed to kolla-ansible (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904853

Revision history for this message

Dr. Jens Harbott (j-harbott) wrote on 2024-01-09:

FTR "openstack image create --import" is the modern version of the glance client command.

I wonder whether the staging directory should also become a docker volume.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-09: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904960
Committed: https://opendev.org/openstack/kolla-ansible/commit/9ecfcf5a17548056eacfa9ba90fc07acb00e64b7
Submitter: "Zuul (22348)"
Branch: master

commit 9ecfcf5a17548056eacfa9ba90fc07acb00e64b7
Author: Michal Arbet <email address hidden>
Date: Mon Jan 8 11:48:35 2024 +0100

Enable glance proxying behaviour

    This patch sets URL to glance worker.
    If this is set, other glance workers will know how to contact this one
    directly if needed. For image import, a single worker stages the image
    and other workers need to be able to proxy the import request to the
    right one.

With current setup glance image import just not working.

Closes-Bug: #2048525

Change-Id: I4246dc8a80038358cd5b6e44e991b3e2ed72be0e

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-12: Fix merged to kolla-ansible (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904850
Committed: https://opendev.org/openstack/kolla-ansible/commit/131296825aecf6d07b21a89d53e58db15d3c1f7b
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 131296825aecf6d07b21a89d53e58db15d3c1f7b
Author: Michal Arbet <email address hidden>
Date: Mon Jan 8 11:48:35 2024 +0100

Enable glance proxying behaviour

With current setup glance image import just not working.

Closes-Bug: #2048525

Change-Id: I4246dc8a80038358cd5b6e44e991b3e2ed72be0e
(cherry picked from commit 9ecfcf5a17548056eacfa9ba90fc07acb00e64b7)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-12: Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904851
Committed: https://opendev.org/openstack/kolla-ansible/commit/106be6dff731e3294189043caa45689b98325d63
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 106be6dff731e3294189043caa45689b98325d63
Author: Michal Arbet <email address hidden>
Date: Mon Jan 8 11:48:35 2024 +0100

Enable glance proxying behaviour

With current setup glance image import just not working.

Closes-Bug: #2048525

Change-Id: I4246dc8a80038358cd5b6e44e991b3e2ed72be0e
(cherry picked from commit 9ecfcf5a17548056eacfa9ba90fc07acb00e64b7)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-12: Fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904852
Committed: https://opendev.org/openstack/kolla-ansible/commit/fd42b97950ce8f6a388f36e6e427f79d80ca03ea
Submitter: "Zuul (22348)"
Branch: stable/zed

commit fd42b97950ce8f6a388f36e6e427f79d80ca03ea
Author: Michal Arbet <email address hidden>
Date: Mon Jan 8 11:48:35 2024 +0100

Enable glance proxying behaviour

With current setup glance image import just not working.

Closes-Bug: #2048525

Change-Id: I4246dc8a80038358cd5b6e44e991b3e2ed72be0e
(cherry picked from commit 9ecfcf5a17548056eacfa9ba90fc07acb00e64b7)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-12: Fix merged to kolla-ansible (stable/yoga)

#10

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904853
Committed: https://opendev.org/openstack/kolla-ansible/commit/d56f46874d207258f83834c3d91425ec940d9fae
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit d56f46874d207258f83834c3d91425ec940d9fae
Author: Michal Arbet <email address hidden>
Date: Mon Jan 8 11:48:35 2024 +0100

Enable glance proxying behaviour

With current setup glance image import just not working.

Closes-Bug: #2048525

Change-Id: I4246dc8a80038358cd5b6e44e991b3e2ed72be0e
(cherry picked from commit 9ecfcf5a17548056eacfa9ba90fc07acb00e64b7)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-25: Fix included in openstack/kolla-ansible 16.3.0

#11

This issue was fixed in the openstack/kolla-ansible 16.3.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-25: Fix included in openstack/kolla-ansible 17.1.0

#12

This issue was fixed in the openstack/kolla-ansible 17.1.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-25: Fix included in openstack/kolla-ansible 14.11.0

#13

This issue was fixed in the openstack/kolla-ansible 14.11.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-25: Fix included in openstack/kolla-ansible 15.4.0

#14

This issue was fixed in the openstack/kolla-ansible 15.4.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-08: Fix included in openstack/kolla-ansible 18.0.0.0rc1

#15

This issue was fixed in the openstack/kolla-ansible 18.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.