WebOb>=1.2.3 requirement for Glance will lead to 0 bytes backing image files on OpenStack Newton, although the image file sent to the python client does not have 0 bytes

[Adrian Vladu]

On CentOS 7 AIO Newton OpenStack deployed with packstack, the default WebOb glance requirement was >= 1.2.3 and pip installed the 1.7.0 version.

When I created a glance image from the cli, using the filesystem backend as default, with a raw file of 1GB, the glance image-create command showed that a image was created, but with the size of 0 bytes.

After forcing with pip the WebOb==1.2.3 version, the issue was not longer there.

I have tried with python 2.7 and 3.4 and to upload the image via horizon or with different python-glance client versions => when WebOb version was 1.7.0 the outcome was wrong, as I ended up with a 0 bytes image in the backing store.

[Ian Cordasco (icordasc)]

Hi Adrian,

This isn't specifically a bug in Glance but in the newest release of WebOb (1.7.0). If you were to install the release just prior to that, everything should work fine. There's no need to install 1.2.3.

This issue has been raised on the mailing list already. It's likely that OpenStack as a whole will blacklist 1.7.0 and 1.7.1 until WebOb can release fixes or until OpenStack has the time to retrofit the APIs on to these new versions.

I'm going to prioritize this with that information in mind.

[Adrian Vladu (avladu)]

Thanks for the info, I thought it was a singular thing.
In this case, backporting the blacklisting is very important.

[Corey Bryant (corey.bryant)]

This is related to issues we're hitting in ubuntu which is at webob 1.7.0 now: https://bugs.launchpad.net/nova/+bug/1657452

[Corey Bryant (corey.bryant)]

Unfortunately this is blocking us from releasing B2 at the moment.

[Ian Cordasco (icordasc)]

So the fundamental part of this is that there's a subtle change in webob 1.7.0 in how webob.request.Request.is_body_readable is implemented.

On 1.7.0, it tells Glance that the body is not readable so it prevents it from reaching certain checks that would produce the appropriate responses. This commit is the root cause of several of Glance's test failures: https://github.com/Pylons/webob/commit/0e6fcbfbc58af6fdd4e125543bf8f5fb91b984c6

[Ian Cordasco (icordasc)]

I filed https://github.com/Pylons/webob/issues/307 although I doubt anything will come of it upstream. If anything, that might help others.

[Corey Bryant (corey.bryant)]

Ian, would you mind if I mark this as a dup of https://bugs.launchpad.net/nova/+bug/1657452 since we're tracking multiple projects there?

[Ian Cordasco (icordasc)]

I don't think it's appropriate to merge them all into one as the impacts are very different as well as the remediation.

[Ian Cordasco (icordasc)]

Another facet of this for Glance seems to be related to the PR in WebOb that broke other projects: http://paste.openstack.org/show/595699/ I suspect this is due to us returning `webob.exc.HTTPNoContent()`. It looks like our gzip middleware expects there to be a content-type but that's no longer auto-generated for us.

[Corey Bryant (corey.bryant)]

Alright, in that case I'll copy your comments over to the other bug. :) Thanks for working on this. Very much appreciate it.

[Corey Bryant (corey.bryant)]

I've lowered the importance from critical to low for Ubuntu. Debian just uploaded python-webob 1:1.6.2-2 to replace 1.7.0-1, which I didn't realize was an option. We'll get 1:1.6.2-2 synced over to zesty and that should solve our webob problems in Ocata.

[Phil Hopkins (phil-hopkins-a)]

Corey,

I think lowering this to low is a mistake, if one does a source install, the way the requirements file is currently set, pip will pull the latest version of WebOb ( a 1.7.?? version) and glance will not upload images (they are all 0 bytes).

The requirements file should be set to block version 1.7 for now.

Phil

[Corey Bryant (corey.bryant)]

Hi Phil, I lowered the glance (Ubuntu) package importance to low, which makes sense since we're locked into a single version for ocata in ubuntu (webob 1.6.2). However for the upstream bug I think your're right. I can't change the importance of the upstream bug, but likely it should be updated.

[Ian Cordasco (icordasc)]

Phil, there is a fix for this. The requirements for Ocata are frozen and will not change, and OpenStack also publishes 'upper-constraints.txt' which you can provide to pip using `-c`. For example

   curl -O https://raw.githubusercontent.com/openstack/requirements/master/upper-constraints.txt
   pip install . -c upper-constraints.txt

WebOb 1.7.x is not in upper-constraints and thus this should solve your problem. It should also ensure your software is more stable as these are the versions OpenStack *actually* tests against.

[Ian Cordasco (icordasc)]

Phil, if you want to test/review the fix you can find it here: https://review.openstack.org/423366

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423366
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=7a843f7e1fb1bab504fea0b2c59bf7c22121da71
Submitter: Jenkins
Branch: master

commit 7a843f7e1fb1bab504fea0b2c59bf7c22121da71
Author: Ian Cordasco <email address hidden>
Date: Fri Jan 20 16:38:10 2017 +0000

    Fix incompatibilities with WebOb 1.7

    WebOb 1.7 changed [0] how request bodies are determined to be
    readable. Prior to version 1.7, the following is how WebOb
    determined if a request body is readable:
      #1 Request method is one of POST, PUT or PATCH
      #2 ``content_length`` length is set
      #3 Special flag ``webob.is_body_readable`` is set

    The special flag ``webob.is_body_readable`` was used to signal
    WebOb to consider a request body readable despite the content length
    not being set. #1 above is how ``chunked`` Transfer Encoding was
    supported implicitly in WebOb < 1.7.

    Now with WebOb 1.7, a request body is considered readable only if
    ``content_length`` is set and it's non-zero [1]. So, we are only left
    with #2 and #3 now. This drops implicit support for ``chunked``
    Transfer Encoding Glance relied on. Hence, to emulate #1, Glance must
    set the the special flag upon checking the HTTP methods that may have
    bodies. This is precisely what this patch attemps to do.

    [0] https://github.com/Pylons/webob/pull/283
    [1] https://github.com/Pylons/webob/pull/283/files#diff-706d71e82f473a3b61d95c2c0d833b60R894

    Closes-bug: #1657459
    Closes-bug: #1657452
    Co-Authored-By: Hemanth Makkapati <email address hidden>
    Change-Id: I19f15165a3d664d5f3a361f29ad7000ba2465a85

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 15.0.0.0b1 development milestone.

[Gurol Akman (gurolakman)]

Hello,

We started to experience this very problem under Ocata running on Ubuntu 17.0.4. At the time of original Ocata installation, all was working fine but then along the way somehow WebOb release has been bumped to 1.7.2 and this problem started to happen.

As a quick-and-dirty workaround, if do the following:

$ pip install WebOb==1.6.0
$ service glance-registry restart
$ service glance-api restart

We can create images properly (i.e., with non-zero size). Note however openstack CLI commands will then start failing as follows:

$ openstack image list
Failed to discover available identity versions when contacting http://controller:35357/v3. Attempting to parse version from URL.
Internal Server Error (HTTP 500)

So, an attempt to switch from WebOb 1.7.2 to 1.6.0 would fix one problem but create another one.

If we want to stay w/ Ocata (which apparently contains glance 14.0.0), what will be the proper steps to fix this problem w/o any side effects similar to the one I tried to describe above.

Many thanks in advance.

[Corey Bryant (corey.bryant)]

Gurol,

If you're installing from Pypi then you need to use the following constraints:
https://github.com/openstack/requirements/blob/stable/ocata/upper-constraints.txt

Otherwise if you're installing from Debian packages, something is wrong. 'apt policy python-webob' may give a hint.

Corey