Glance download with allowed_direct_url_schemes sometimes fails with Python 3

[MarkMielke]

This is similar to 1775678 and a few others. The code in GlanceImageService.download() (cinder/image/glance.py) specifies:

with open(parsed_url.path, "r") as f:
    shutil.copyfileobj(f, data)

This code only takes effect if "file" is specified for allowed_direct_url_schemes.

This results in hard to debug character set errors during image download. Luckily, it seems that the image usually fails to download with an obscure error rather than becoming corrupt.

This should have been caught with the move to Python 3. Python 3 requires "rb" when opening binary files. "r" opens in text mode. It may be that allowed_direct_url_scheme is not used by very many people. However, I still use it, and I have been holding onto a local patch ("r" => "rb") since Train which resolves the problem every time. Opening this issue to start the process of getting the fix posted.

[Eric Harney (eharney)]

https://review.opendev.org/c/openstack/cinder/+/785878

[Brian Rosmaita (brian-rosmaita)]

@Mark: Thanks for catching this and for the detailed report!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/786922

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/785878
Committed: https://opendev.org/openstack/cinder/commit/5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Submitter: "Zuul (22348)"
Branch: master

commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

    Open local image files with "rb" mode

    When using "file" in allowed_direct_url_schemes,
    open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/786922
Committed: https://opendev.org/openstack/cinder/commit/c16567bd8784761d24b73678377994903b8dc35b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c16567bd8784761d24b73678377994903b8dc35b
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

    Open local image files with "rb" mode

    When using "file" in allowed_direct_url_schemes,
    open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42
    (cherry picked from commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/793169

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/793169
Committed: https://opendev.org/openstack/cinder/commit/f6d7b09195ce738576252736708a7f3417f761db
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f6d7b09195ce738576252736708a7f3417f761db
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

    Open local image files with "rb" mode

    When using "file" in allowed_direct_url_schemes,
    open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42
    (cherry picked from commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc)
    (cherry picked from commit c16567bd8784761d24b73678377994903b8dc35b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 19.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 16.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 18.1.0 release.