node_staging_uri needs the file store to be configured

[Thomas Herve]

If you set node_staging_uri=file:///var/lib/glance/staging and try to do a web-download import without the file store enabled, this is logged:

 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
     result = task.execute(**arguments)
   File "/usr/lib/python2.7/site-packages/glance/async_/flows/api_image_import.py", line 92, in execute
     store_api.delete_from_backend(file_path)
   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 409, in delete_from_backend
     loc = location.get_location_from_uri(uri, conf=CONF)
   File "/usr/lib/python2.7/site-packages/glance_store/location.py", line 75, in get_location_from_uri
     raise exceptions.UnknownScheme(scheme=pieces.scheme)
 UnknownScheme: Unknown scheme 'file' found in URI

The consequence is that the staging area is not cleaned up, and files are piling up. The store should probably be automatically configured.

[Abhishek Kekane (abhishek-kekane)]

This is the case when operator has not enabled file store, i.e. he is using other stores like ceph, swift etc. In this situation, the uploading to staging area works as we explicitly build file store during this operation, while cleaning up we directly use glance_store.delete_from_backend which only works if 'file' store is enabled by operator explicitly.

This is not user-friendly, what I recommend for upstream fix is;

1. Document that in case if you want to use image-import api then 'file' store needs to be in the list of 'stores' defined in glance-api.conf

2. We can load file store at [1] and use store.delete method directly there.

[1] https://github.com/openstack/glance/blob/master/glance/async_/flows/api_image_import.py#L78

NOTE:
Once multiple backend is stabilized we will not face this issue as we will be reserving a file store for this purpose. (refer, https://review.openstack.org/605584)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/618468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618468
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=c92724608512d42b56152d199cc0fea049e4d4a7
Submitter: Zuul
Branch: master

commit c92724608512d42b56152d199cc0fea049e4d4a7
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 13 18:12:42 2019 +0000

    Data remains in staging area if 'file' store is not enabled

    When operator has not enabled 'file' store and using other stores like ceph,
    swift etc. the uploading to staging area works as we explicitly
    build 'file' store during this operation, while cleaning up we directly
    use 'glance_store.delete_from_backend' which only works if 'file'
    store is enabled.

    Modified '_DeleteFromFS' task and _unstage call which will use os
    module to unlink the file present in staging area explicitly to
    delete the data from staging area.

    Closes-Bug: #1803498
    Change-Id: If0b3b0af9300301291758c67267890e0959ebb3c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/644776

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/644776
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=8c3751db21133fde4e47965ac6ea9971b3352c50
Submitter: Zuul
Branch: stable/rocky

commit 8c3751db21133fde4e47965ac6ea9971b3352c50
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 13 18:12:42 2019 +0000

    Data remains in staging area if 'file' store is not enabled

    When operator has not enabled 'file' store and using other stores like ceph,
    swift etc. the uploading to staging area works as we explicitly
    build 'file' store during this operation, while cleaning up we directly
    use 'glance_store.delete_from_backend' which only works if 'file'
    store is enabled.

    Modified '_DeleteFromFS' task and _unstage call which will use os
    module to unlink the file present in staging area explicitly to
    delete the data from staging area.

    Conflicts:
          glance/tests/unit/async/flows/test_web_download.py

    NOTE: Renamed import statement from above test files as async_
    module is not available in stable/rocky.

    Closes-Bug: #1803498
    Change-Id: If0b3b0af9300301291758c67267890e0959ebb3c
    (cherry picked from commit c92724608512d42b56152d199cc0fea049e4d4a7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/648610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648610
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=14e9e1ef868759df2be67f7b61ce3c41ec547ac2
Submitter: Zuul
Branch: stable/queens

commit 14e9e1ef868759df2be67f7b61ce3c41ec547ac2
Author: Abhishek Kekane <email address hidden>
Date: Wed Mar 13 18:12:42 2019 +0000

    Data remains in staging area if 'file' store is not enabled

    When operator has not enabled 'file' store and using other stores like ceph,
    swift etc. the uploading to staging area works as we explicitly
    build 'file' store during this operation, while cleaning up we directly
    use 'glance_store.delete_from_backend' which only works if 'file'
    store is enabled.

    Modified '_DeleteFromFS' task and _unstage call which will use os
    module to unlink the file present in staging area explicitly to
    delete the data from staging area.

    Conflicts:
           glance/async/flows/api_image_import.py
           glance/tests/unit/async_/flows/test_web_download.py

    NOTE: async_ module was not available in queens so renamed
    the module name to async.

    Closes-Bug: #1803498
    Change-Id: If0b3b0af9300301291758c67267890e0959ebb3c
    (cherry picked from commit c92724608512d42b56152d199cc0fea049e4d4a7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 17.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance queens-eol release.