Glance download with allowed_direct_url_schemes sometimes fails with Python 3

Bug #1923313 reported by MarkMielke on 2021-04-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Status tracked in Xena
Train
Undecided
Unassigned
Ussuri
Undecided
Unassigned
Victoria
Undecided
Unassigned
Wallaby
Undecided
Unassigned
Xena
High
Eric Harney

Bug Description

This is similar to 1775678 and a few others. The code in GlanceImageService.download() (cinder/image/glance.py) specifies:

with open(parsed_url.path, "r") as f:
    shutil.copyfileobj(f, data)

This code only takes effect if "file" is specified for allowed_direct_url_schemes.

This results in hard to debug character set errors during image download. Luckily, it seems that the image usually fails to download with an obscure error rather than becoming corrupt.

This should have been caught with the move to Python 3. Python 3 requires "rb" when opening binary files. "r" opens in text mode. It may be that allowed_direct_url_scheme is not used by very many people. However, I still use it, and I have been holding onto a local patch ("r" => "rb") since Train which resolves the problem every time. Opening this issue to start the process of getting the fix posted.

MarkMielke (mark-mielke) on 2021-04-10
description: updated
Revision history for this message
Eric Harney (eharney) wrote :
Changed in cinder:
status: New → In Progress
assignee: nobody → Eric Harney (eharney)
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

@Mark: Thanks for catching this and for the detailed report!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/786922

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/785878
Committed: https://opendev.org/openstack/cinder/commit/5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Submitter: "Zuul (22348)"
Branch: master

commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

    Open local image files with "rb" mode

    When using "file" in allowed_direct_url_schemes,
    open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/786922
Committed: https://opendev.org/openstack/cinder/commit/c16567bd8784761d24b73678377994903b8dc35b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c16567bd8784761d24b73678377994903b8dc35b
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

    Open local image files with "rb" mode

    When using "file" in allowed_direct_url_schemes,
    open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42
    (cherry picked from commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/793169

tags: added: files glance python3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers