failed glance image download leads to NoneType error

[Marc Methot]

This ERROR should be handled prior to trying to iteration attempt.

2018-09-28 11:38:10.124 3320 ERROR cinder.scheduler.filter_scheduler [req-e8cb2375-a6d3-4afb-a146-21be0d1c8777 d6c1ad4f37c74ffd862f5c22cc869ce5 0896f504b30e4522b605d4f3c784dca1 - default default] Error scheduling None from last vol-service: hostgroup@scaleio#OPS:Standard : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 951, in execute\n **volume_spec)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 898, in _create_from_image\n image_service)\n', u' File "<string>", line 2, in _create_from_image_cache_or_download\n', u' File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 176, in _synchronized\n return f(*a, **k)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 779, in _create_from_image_cache_or_download\n backend_name) as tmp_image:\n', u' File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__\n return self.gen.next()\n', u' File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 670, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', u' File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 288, in fetch_verify_image\n None, None)\n', u' File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 223, in fetch\n image_service.download(context, image_id, image_file)\n', u' File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 341, in download\n for chunk in image_chunks:\n', u"TypeError: 'NoneType' object is not iterable\n"]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611081

[Marc Methot (mb-methot)]

The glance documentation states that None (empty image) is a valid return value. We have to handle this.
https://docs.openstack.org/python-glanceclient/latest/reference/api/glanceclient.v2.images.html#glanceclient.v2.images.Controller.data

"Returns: An iterable body or None"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611081
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d3afc39467c6532344118e8e491409878da9c2dd
Submitter: Zuul
Branch: master

commit d3afc39467c6532344118e8e491409878da9c2dd
Author: Marc Methot <email address hidden>
Date: Tue Oct 16 12:37:28 2018 -0400

    Handling unexpected python error "NoneType object is not iterable"

    The glance documentation states that None (empty image) is a valid
    return value. We have to handle this outcome. Adding error handling
    helps debugging this unhandled issue

    Change-Id: If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138
    Closes-Bug: #1798147

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/618978

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618978
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=805368e902ada4c63e2e6e4d6692cf8cc2531a1e
Submitter: Zuul
Branch: stable/rocky

commit 805368e902ada4c63e2e6e4d6692cf8cc2531a1e
Author: Marc Methot <email address hidden>
Date: Tue Oct 16 12:37:28 2018 -0400

    Handling unexpected python error "NoneType object is not iterable"

    The glance documentation states that None (empty image) is a valid
    return value. We have to handle this outcome. Adding error handling
    helps debugging this unhandled issue

    Change-Id: If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138
    Closes-Bug: #1798147
    (cherry picked from commit d3afc39467c6532344118e8e491409878da9c2dd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/629463
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=937af5be0e7c17b34860e25cc434a219d7143387
Submitter: Zuul
Branch: stable/queens

commit 937af5be0e7c17b34860e25cc434a219d7143387
Author: Francois Deppierraz <email address hidden>
Date: Mon Oct 22 15:33:25 2018 +0200

    cinder-volume: Stop masking IOError different than ENOSPC

    When glanceclient raises an IOError with a different errno than ENOSPC,
    cinder-volume silently masked it and continued its volume creation
    process. The result was volumes with invalid content being successfuly
    created.

    With the patch, an ImageDownloadFailed exception is raised in this case,
    which makes the volume creation process fail and gives enough
    information to operators for troubleshooting.

    As explained in detail below, this patch is a squash of four cherry
    picks to fix Bug #1799221. The cherry-picks are being squashed
    instead of done separately per normal cinder practice because they
    are dependent; this will make sure that one of them isn't further
    backported without the supporting patches.

    Here's the relationship between the four cherry-picks:
    (1) The purpose of this backport is to fix Bug #1799221, which was
        introduced in Pike. It is fixed in Stein and Rocky by
        Ic011fe30b4840e5098db1a594ea276ec98768bff
    (2) That change requires an exception introduced in Stein and backported
        to Rocky by If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138 to fix Bug
        #1798147 (which defect is also present in Queens and Pike)
    (3) The change in (2) introduced Bug #1808443 which was fixed in Stein
        and Rocky by I6d8dedfd056add3414f8f4bf7f7279eae4763286
    (4) The change in (2) also introduced Bug #1811184, which is fixed by
        I6d8dedfd056add3414f8f4bf7f7279eae4763286 in Stein and Rocky, and
        which adds a unit test for Bug #1798147.
    In short, in order to backport the fix for (1), we need to backport (2),
    but in order to backport (2) we need to follow up immediately with
    backports of (3) and (4) to fix the defects (2) introduces.

    The attentive reader will note that this patch smuggles in the fix for
    Bug #1798147. We could have left this out, but it's a very small
    isolated change, the defect is present in Queens and Pike (remember that
    Bug #1799221, the subject of this patch, was introduced in Pike), and it
    has a unit test (see (4), above). Finally, leaving out the fix for
    Bug #1798147 and backporting only the exception would still require
    backporting the fixes for Bug #1808443 and Bug #1811184, so it really
    would not simplify this patch.

    To summarize what's being included here:
    commit changeId fixed bug
    805368e If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138 Bug #1798147
    9c696ce I2aa56da73660794c6dedcbb8a66e84bcec511a9c Bug #1808443
    844b627 I6d8dedfd056add3414f8f4bf7f7279eae4763286 Bug #1811184
    bf89f76 Ic011fe30b4840e5098db1a594ea276ec98768bff Bug #1799221

    Change-Id: Ic011fe30b4840e5098db1a594ea276ec98768bf...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/633015

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633015
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0551f0a9ff7b787fcb3e1b686b83e25f99cad874
Submitter: Zuul
Branch: stable/pike

commit 0551f0a9ff7b787fcb3e1b686b83e25f99cad874
Author: Francois Deppierraz <email address hidden>
Date: Mon Oct 22 15:33:25 2018 +0200

    cinder-volume: Stop masking IOError different than ENOSPC

    When glanceclient raises an IOError with a different errno than ENOSPC,
    cinder-volume silently masked it and continued its volume creation
    process. The result was volumes with invalid content being successfuly
    created.

    With the patch, an ImageDownloadFailed exception is raised in this case,
    which makes the volume creation process fail and gives enough
    information to operators for troubleshooting.

    As explained in detail below, this patch is a squash of four cherry
    picks to fix Bug #1799221. The cherry-picks are being squashed
    instead of done separately per normal cinder practice because they
    are dependent; this will make sure that one of them isn't further
    backported without the supporting patches.

    Here's the relationship between the four cherry-picks:
    (1) The purpose of this backport is to fix Bug #1799221, which was
        introduced in Pike. It is fixed in Stein and Rocky by
        Ic011fe30b4840e5098db1a594ea276ec98768bff
    (2) That change requires an exception introduced in Stein and backported
        to Rocky by If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138 to fix Bug
        #1798147 (which defect is also present in Queens and Pike)
    (3) The change in (2) introduced Bug #1808443 which was fixed in Stein
        and Rocky by I6d8dedfd056add3414f8f4bf7f7279eae4763286
    (4) The change in (2) also introduced Bug #1811184, which is fixed by
        I6d8dedfd056add3414f8f4bf7f7279eae4763286 in Stein and Rocky, and
        which adds a unit test for Bug #1798147.
    In short, in order to backport the fix for (1), we need to backport (2),
    but in order to backport (2) we need to follow up immediately with
    backports of (3) and (4) to fix the defects (2) introduces.

    The attentive reader will note that this patch smuggles in the fix for
    Bug #1798147. We could have left this out, but it's a very small
    isolated change, the defect is present in Queens and Pike (remember that
    Bug #1799221, the subject of this patch, was introduced in Pike), and it
    has a unit test (see (4), above). Finally, leaving out the fix for
    Bug #1798147 and backporting only the exception would still require
    backporting the fixes for Bug #1808443 and Bug #1811184, so it really
    would not simplify this patch.

    To summarize what's being included here:
    commit changeId fixed bug
    805368e If7c22ac4516f8c2a6ccd8bf6b6ed98409312b138 Bug #1798147
    9c696ce I2aa56da73660794c6dedcbb8a66e84bcec511a9c Bug #1808443
    844b627 I6d8dedfd056add3414f8f4bf7f7279eae4763286 Bug #1811184
    bf89f76 Ic011fe30b4840e5098db1a594ea276ec98768bff Bug #1799221

    Change-Id: Ic011fe30b4840e5098db1a594ea276ec98768bff
...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/644954

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/644954