[Swift backend] Upload image hit error: Unicode-objects must be encoded before hashing

[wangxiyuan]

[Impact]
If we upload a large image (larger than 1G), the glance_store will hit a Unicode error. To fix this a patch has been merged in upstream master and backported to stable rocky.

[Test Case]
Deploy glance related to swift-proxy using the object-store relation. Then attempt to upload a large image (not cirros)

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

If the patch is manually applied and glance-api restarted then the above command succeeds.

In order to avoid regression of existing consumers, the OpenStack team will
run their continuous integration test against the packages that are in
-proposed. A successful run of all available tests will be required before the
proposed packages can be let into -updates.

The OpenStack team will be in charge of attaching the output summary of the
executed tests. The OpenStack team members will not mark ‘verification-done’ until
this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned tests are attached to this bug.

[Discussion]
n/a

[Original Description]

env: master branch, Glance using swift backend.

We hit a strange error， if we upload a large image (larger than 1G), the glance_store will hit a error:Unicode-objects must be encoded before hashing. But if the image is small enough, the error won't happen.

error log:
https://www.irccloud.com/pastebin/jP3DapNy/

After dig into the code, it appears that when chunk reading the image data, the date piece may be non-byte, so the checksum.updating will raise the error.

encoding the date piece to ensure it's byte can solve the problem.

[wangxiyuan (wangxiyuan)]

https://review.openstack.org/#/c/620234/

[Brian Rosmaita (brian-rosmaita)]

I really didn't like the utf-8 encoding, because if there's a byte with a value over 127, it will be encoded as 2 bytes. As far as we can tell, it looks like this is happening when a zero-byte read is requested (bytes!! so why are we getting a unicode object??). See this etherpad for more info: https://etherpad.openstack.org/p/glance_store-py3-swift-driver-problem

[Brian Rosmaita (brian-rosmaita)]

timburke in IRC noticed this: https://github.com/openstack/glance_store/blob/0.28.0/glance_store/common/utils.py#L138

Looks like that will cause the same problem for anything that uses the CooperativeReader.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620234
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=1d25a2b7a21e95766f9fee378b3d0802d392a85f
Submitter: Zuul
Branch: master

commit 1d25a2b7a21e95766f9fee378b3d0802d392a85f
Author: wangxiyuan <email address hidden>
Date: Tue Nov 27 14:50:50 2018 +0800

    Prevent unicode object error from zero-byte read

    During large file uploads under py3, we are occasionally seeing a
    "unicode objects must be encoded before hashing" error even though
    we are reading from a byte stream. From what I can tell, it looks
    like it's happening when a zero-byte read is requested, so we handle
    that case explicitly. This is a band-aid fix; we still need to track
    down the source.

    Co-authored-by: wangxiyuan <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Related-bug: #1805332
    Change-Id: Ia7653f9fcbe902abc203c10c80ab44a641a4d8f9

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/644839

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/644895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/644839
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=9c8364bacfbe831a755b096a92fb7da2ff3c878d
Submitter: Zuul
Branch: stable/stein

commit 9c8364bacfbe831a755b096a92fb7da2ff3c878d
Author: wangxiyuan <email address hidden>
Date: Tue Nov 27 14:50:50 2018 +0800

    Prevent unicode object error from zero-byte read

    During large file uploads under py3, we are occasionally seeing a
    "unicode objects must be encoded before hashing" error even though
    we are reading from a byte stream. From what I can tell, it looks
    like it's happening when a zero-byte read is requested, so we handle
    that case explicitly. This is a band-aid fix; we still need to track
    down the source.

    Co-authored-by: wangxiyuan <email address hidden>
    Co-authored-by: Brian Rosmaita <email address hidden>

    Related-bug: #1805332
    Change-Id: Ia7653f9fcbe902abc203c10c80ab44a641a4d8f9
    (cherry picked from commit 1d25a2b7a21e95766f9fee378b3d0802d392a85f)

[Sahid Orentino (sahid-ferdjaoui)]

[Impact]
If we upload a large image (larger than 1G), the glance_store will hit a Unicode error. To fix this a patch has been merged in upstream master and backported to stable rocky.

[Test Case]
In order to avoid regression of existing consumers, the OpenStack team will
run their continuous integration test against the packages that are in
-proposed. A successful run of all available tests will be required before the
proposed packages can be let into -updates.

The OpenStack team will be in charge of attaching the output summary of the
executed tests. The OpenStack team members will not mark ‘verification-done’ until
this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned tests are attached to this bug.

[Discussion]
n/a

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 0.28.0-0ubuntu2

---------------
python-glance-store (0.28.0-0ubuntu2) eoan; urgency=medium

  * d/p/prevent-unicode-object-error-from-zero-byte-read: Fix unicode
    issue (LP: #1805332).

 -- Sahid Orentino Ferdjaoui <email address hidden> Wed, 24 Apr 2019 13:46:56 +0100

[Steve Langasek (vorlon)]

This bug is missing an SRU template.

[Łukasz Zemczak (sil2100)]

SRU template is available in comment #8. Moved it into the description and proceeding with upload review.

[Łukasz Zemczak (sil2100)]

An upload of python-glance-store to disco-proposed has been rejected from the upload queue for the following reason: "SRU test case seems a bit insufficient: it is a very good idea to request a full CI run for the packages, but it doesn't feel sufficient for checking if the target bug is fixed. The added patch doesn't come with any tests, so the upload should come with a test-case trying to test if the actual regression has been fixed. Also, simply wondering: the cherry-picked fix has been named a 'band-aid fix', with the underlying issue still needing fixing - is there still no full fix for this?".

[Liam Young (gnuoy)]

It does not appear to have been fixed upstream yet as this patch is still in place at master: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L1635

[Łukasz Zemczak (sil2100)]

Hello wangxiyuan, or anyone else affected,

Accepted python-glance-store into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/0.28.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello wangxiyuan, or anyone else affected,

Accepted python-glance-store into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Liam Young (gnuoy)]

Looks good to me. Tested 0.28.0-0ubuntu1.1~cloud0 from cloud-archive:stein-proposed

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

$ juju run --unit glance/0 "add-apt-repository cloud-archive:stein-proposed --yes --update"
Reading package lists...
Building dependency tree...
Reading state information...
ubuntu-cloud-keyring is already the newest version (2018.09.18.1~18.04.0).
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://nova.clouds.archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Ign:4 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/stein InRelease ...

[Liam Young (gnuoy)]

The disco package worked for me to. Version info below:

# apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 0.28.0-0ubuntu1.1
  Candidate: 0.28.0-0ubuntu1.1
  Version table:
 *** 0.28.0-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.28.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco/main amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 0.28.0-0ubuntu1.1

---------------
python-glance-store (0.28.0-0ubuntu1.1) disco; urgency=medium

  * d/p/prevent-unicode-object-error-from-zero-byte-read: Fix unicode
    issue (LP: #1805332).

 -- Sahid Orentino Ferdjaoui <email address hidden> Wed, 24 Apr 2019 13:46:43 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Brian Murray (brian-murray)]

Hello wangxiyuan, or anyone else affected,

Accepted python-glance-store into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/0.26.1-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 0.28.0-0ubuntu1.1~cloud0
---------------

 python-glance-store (0.28.0-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (0.28.0-0ubuntu1.1) disco; urgency=medium
 .
   * d/p/prevent-unicode-object-error-from-zero-byte-read: Fix unicode
     issue (LP: #1805332).

[Corey Bryant (corey.bryant)]

Hello wangxiyuan, or anyone else affected,

Accepted python-glance-store into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Liam Young (gnuoy)]

The cosmic package worked for me to. Version info below:

python3-glance-store:
  Installed: 0.26.1-0ubuntu2.1
  Candidate: 0.26.1-0ubuntu2.1
  Version table:
 *** 0.26.1-0ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.26.1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages

Test output:

$ juju run --unit glance/0 "apt-cache policy python3-glance-store"
python3-glance-store:
  Installed: 0.26.1-0ubuntu2
  Candidate: 0.26.1-0ubuntu2
  Version table:
 *** 0.26.1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages
        100 /var/lib/dpkg/status

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

* Enable proposed, upgrade python3-glance-store and restart glance-api service

$ juju run --unit glance/0 "apt-cache policy python3-glance-store"
python3-glance-store:
  Installed: 0.26.1-0ubuntu2.1
  Candidate: 0.26.1-0ubuntu2.1
  Version table:
 *** 0.26.1-0ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu cosmic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.26.1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | c8994590c7d61dc68922e461686ef936 |
| container_format | bare |
| created_at | 2019-05-22T07:10:26Z |
| disk_format | raw |
| file | /v2/images/eca7aeb5-4c16-4bb2-ad9a-53acfb3c18ca/file ...

[Liam Young (gnuoy)]

The package from rocky-proposed worked for me. Version info below:
python3-glance-store:
  Installed: 0.26.1-0ubuntu2.1~cloud0
  Candidate: 0.26.1-0ubuntu2.1~cloud0
  Version table:
 *** 0.26.1-0ubuntu2.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     0.26.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
     0.23.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

Test output:

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

$ juju run --unit glance/0 "apt-cache policy python3-glance-store"
python3-glance-store:
  Installed: 0.26.1-0ubuntu2~cloud0
  Candidate: 0.26.1-0ubuntu2~cloud0
  Version table:
 *** 0.26.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     0.23.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

$ juju run --unit glance/0 "add-apt-repository cloud-archive:rocky-proposed --yes --update"
...
$ juju run --unit glance/0 "apt install --yes python3-glance-store; systemctl restart glance-api"
...
(clients) ubuntu@gnuoy-bastion2:~/branches/nova-compute$ juju run --unit glance/0 "apt-cache policy python3-glance-store"
python3-glance-store:
  Installed: 0.26.1-0ubuntu2.1~cloud0
  Candidate: 0.26.1-0ubuntu2.1~cloud0
  Version table:
 *** 0.26.1-0ubuntu2.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     0.26.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
     0.23.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

$ openstack image create --public --file /home/ubuntu/images/bionic-server-cloudimg-amd64.img bionic-test
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | c8994590c7d61dc68922e461686ef936 |
|...

[Robie Basak (racb)]

14:09 <rbasak> coreycb: have you done a CI run against python-glance-store as well as having verified the individual fix? I don't see any CI report in the bug.

14:11 <coreycb> rbasak: i've not but we generally only run CI for stable point releases. i certainly can though.

14:17 <rbasak> coreycb: the bug description said you'd run it! I don't mind how it's achieved but I'd prefer to have some confidence that the SRU doesn't break existing functionality, and CI seems like a good way to do that.

14:17 <rbasak> coreycb: especially if the bug being fixed is particularly targeted and so the test to verify the bug fix itself doesn't exercise much of the package, which I presume here it doesn't?

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 0.26.1-0ubuntu2.1

---------------
python-glance-store (0.26.1-0ubuntu2.1) cosmic; urgency=medium

  [ Sahid Orentino Ferdjaoui ]
  * d/p/prevent-unicode-object-error-from-zero-byte-read.patch: Fix unicode
    issue (LP: #1805332)

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/rocky branch.

 -- Sahid Orentino Ferdjaoui <email address hidden> Wed, 24 Apr 2019 13:46:29 +0100

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 0.26.1-0ubuntu2.1~cloud0
---------------

 python-glance-store (0.26.1-0ubuntu2.1~cloud0) bionic-rocky; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (0.26.1-0ubuntu2.1) cosmic; urgency=medium
 .
   [ Sahid Orentino Ferdjaoui ]
   * d/p/prevent-unicode-object-error-from-zero-byte-read.patch: Fix unicode
     issue (LP: #1805332)
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/rocky branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/644895
Reason: I haven't had time to follow up on this.