Image remains in queued state for web-download when node_staging_uri end with "/"

[Abhishek Kekane]

The node_staging_uri has a default of file:///tmp/staging/
see: https://github.com/openstack/glance/blob/a0aaa614712090e7cab19dc35a155c32ea8f2190/glance/common/config.py#L680

If operator does not set 'node_staging_uri' in glance-api.conf then importing image using web-download remains in queued state.

Steps to reproduce:
1. Ensure glance-api is running under mod_wsgi (add WSGI_MODE=mod_wsgi in local.conf and run stack.sh)
2. Do not set node_staging_uri in glance-api.conf

3. Create image using below curl command:
curl -i -X POST -H "x-auth-token: <auth-token>" http://192.168.0.13:9292/v2/images -d '{"container_format":"bare","disk_format":"raw","name":"Import web-download"}'

4. Import image using below curl command:
curl -i -X POST -H "Content-type: application/json" -H "x-auth-token: <auth-token>" http://192.168.0.13:9292/v2/images/<id-of-image-created-in-step-3>/import -d '{"method":{"name":"web-download","uri":"https://www.openstack.org/assets/openstack-logo/2016R/OpenStack-Logo-Horizontal.eps.zip"}}'

Expected result:
Image should be in active state.

Actual result:
Image remains in queued state.

API Logs:
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: DEBUG glance_store.backend [-] Attempting to import store file {{(pid=3506) _load_store /usr/local/lib/python2.7/dist-packages/glance_store/backend.py:231}}
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: DEBUG glance_store.capabilities [-] Store glance_store._drivers.filesystem.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. {{(pid=3506) update_capabilities /usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py:97}}
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: Traceback (most recent call last):
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: func(*args, **kwargs)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/notifier.py", line 581, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: super(TaskProxy, self).run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/proxy.py", line 238, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self.base.run(executor)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/domain/__init__.py", line 438, in run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: executor.begin_processing(self.task_id)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 144, in begin_processing
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: super(TaskExecutor, self).begin_processing(task_id)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/__init__.py", line 63, in begin_processing
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self._run(task_id, task.type)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 165, in _run
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: flow = self._get_flow(task)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/taskflow_executor.py", line 134, in _get_flow
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: invoke_kwds=kwds).driver
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 61, in __init__
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: warn_on_missing_entrypoint=warn_on_missing_entrypoint
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 81, in __init__
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 203, in _load_plugins
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: self._on_load_failure_callback(self, ep, err)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 195, in _load_plugins
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements,
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: verify_requirements,
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 227, in _load_one_plugin
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: obj = plugin(*invoke_args, **invoke_kwds)
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: File "/opt/stack/glance/glance/async/flows/api_image_import.py", line 323, in get_flow
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: file_uri = separator.join((CONF.node_staging_uri, str(image_id)))
Mar 07 09:26:07 ubuntu-16 glance-api[3499]: UnboundLocalError: local variable 'separator' referenced before assignment

[Abhishek Kekane (abhishek-kekane)]

Fix proposed to Master :-> https://review.openstack.org/#/c/550417/1

[Erno Kuvaja (jokke)]

So, the actual bug is invalid, there is clearly a bug in the side effect though, which is it throwing 500.

The node_staging_uri is used as local cache for the taskflow to store the data it downloads before the rest of the tasks are ran. So it is needed for the 'web-download' (In 'web-download' only use case it does not need to be shared between the glance nodes). If the methods are incorrectly configured, the image should stay in 'queued' but we should fail the import call gracefully.

[Brian Rosmaita (brian-rosmaita)]

The problem isn't that the operator has configured None for node_staging_uri ... the problem occurs when the operator runs with the default value, which is 'file:///tmp/staging/', and someone tries a web-download import.

This is actually a bad bug. I think the normal way to write a directory is with a '/' at the end, and doing so will cause this failure to occur.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550417
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c
Submitter: Zuul
Branch: master

commit ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 7 09:56:50 2018 +0000

    Fix web-download fails with default node_staging_uri

    Web-download import method fails to import image to backend if
    'node_staging_uri' is not set in glance-api.conf (or when the value
    ends with a '/', as the default setting does) due to "local variable
    'separator' referenced before assignment" error.

    Co-authored-by: Abhishek Kekane <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Change-Id: I056fb7a1f7a7ef6bc14ea54133cf083b2b9d9920
    Closes-Bug: #1753964

[Brian Rosmaita (brian-rosmaita)]

Change proposed to stable/queens: https://review.openstack.org/#/c/559073/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/559073
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=40c886662e4cc6a633b858978b702b7fc257a579
Submitter: Zuul
Branch: stable/queens

commit 40c886662e4cc6a633b858978b702b7fc257a579
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 7 09:56:50 2018 +0000

    Fix web-download fails with default node_staging_uri

    Web-download import method fails to import image to backend if
    'node_staging_uri' is not set in glance-api.conf (or when the value
    ends with a '/', as the default setting does) due to "local variable
    'separator' referenced before assignment" error.

    Co-authored-by: Abhishek Kekane <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Change-Id: I056fb7a1f7a7ef6bc14ea54133cf083b2b9d9920
    Closes-Bug: #1753964
    (cherry picked from commit ff1ba4f3f5afcbd06a7bbb4cf4363ac8f368ee4c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 16.0.1 release.