TestRequestBodySizeLimiter.test_request_too_large_no_content_length fails with low concurrency

[Thomas Bechtold]

When running tox, the test oslo_middleware.tests.test_sizelimit.TestRequestBodySizeLimiter.test_request_too_large_no_content_length fails.

Steps to reproduce:

$ tox -epy27 --oslo_middleware.tests.test_sizelimit.TestRequestBodySizeLimiter.test_request_too_large_no_content_length

GLOB sdist-make: /home/tom/devel/openstack/oslo.middleware/setup.py
py27 create: /home/tom/devel/openstack/oslo.middleware/.tox/py27
py27 installdeps: -r/home/tom/devel/openstack/oslo.middleware/test-requirements.txt
py27 inst: /home/tom/devel/openstack/oslo.middleware/.tox/dist/oslo.middleware-3.25.0.zip
py27 installed: alabaster==0.7.10,appdirs==1.4.3,Babel==2.4.0,coverage==4.3.4,debtcollector==1.13.0,docutils==0.13.1,dulwich==0.17.3,extras==1.0.0,fixtures==3.0.0,flake8==2.5.5,funcsigs==1.0.2,hacking==0.12.0,imagesize==0.7.1,iso8601==0.1.11,Jinja2==2.9.6,keystoneauth1==2.19.0,linecache2==1.0.0,MarkupSafe==1.0,mccabe==0.2.1,mock==2.0.0,monotonic==1.3,mox3==0.21.0,netaddr==0.7.19,netifaces==0.10.5,openstack-requirements==1.0.0,os-client-config==1.26.0,oslo.config==3.24.0,oslo.context==2.13.0,oslo.i18n==3.15.0,oslo.middleware==3.25.0,oslo.utils==3.25.0,oslosphinx==4.11.0,oslotest==2.15.0,packaging==16.8,Parsley==1.3,pbr==2.0.0,pep8==1.5.7,positional==1.1.1,pyflakes==0.8.1,Pygments==2.2.0,pyparsing==2.2.0,python-mimeparse==1.6.0,python-subunit==1.2.0,pytz==2017.2,PyYAML==3.12,reno==2.2.0,requests==2.12.5,requestsexceptions==1.2.0,rfc3986==0.4.1,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,statsd==3.2.1,stevedore==1.21.0,testrepository==0.0.20,testtools==2.2.0,traceback2==1.4.0,unittest2==1.1.0,WebOb==1.7.2,wrapt==1.10.10
py27 runtests: PYTHONHASHSEED='2531496026'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=--parallel --concurrency 2 oslo_middleware.tests.test_sizelimit.TestRequestBodySizeLimiter.test_request_too_large_no_content_length
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
${PYTHON:-python} -m subunit.run discover -t ./ . --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpwN5ASs
======================================================================
FAIL: oslo_middleware.tests.test_sizelimit.TestRequestBodySizeLimiter.test_request_too_large_no_content_length
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "oslo_middleware/tests/test_sizelimit.py", line 108, in test_request_too_large_no_content_length
    self.assertEqual(413, response.status_int)
  File "/home/tom/devel/openstack/oslo.middleware/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/tom/devel/openstack/oslo.middleware/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 413 != 200
Ran 1 tests in 0.003s
FAILED (id=0, failures=1)
error: testr failed (1)
ERROR: InvocationError: '/home/tom/devel/openstack/oslo.middleware/.tox/py27/bin/python setup.py testr --slowest --testr-args=--parallel --concurrency 2 oslo_middleware.tests.test_sizelimit.TestRequestBodySizeLimiter.test_request_too_large_no_content_length'
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR: py27: commands failed

[Thomas Bechtold (toabctl)]

The problem seems to be WebOb related. I guess commit https://github.com/Pylons/webob/commit/0e6fcbfbc58af6fdd4e125543bf8f5fb91b984c6 is the reason.
The is_body_readable() method returns False now (in no content length is set) because WebOb removed http_method_probably_has_body which returned True in our test case (which uses POST)

[ChangBo Guo(gcb) (glongwave)]

It seems WebOb >=1.7.0 changes behavior, oslo.middleware should handle the different behavior.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/453712

[ChangBo Guo(gcb) (glongwave)]

This is also related with https://bugs.launchpad.net/oslo.middleware/+bug/1657452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453712
Committed: https://git.openstack.org/cgit/openstack/oslo.middleware/commit/?id=fed29257358f990dd91eccd9be08e4cbcea53dca
Submitter: Jenkins
Branch: master

commit fed29257358f990dd91eccd9be08e4cbcea53dca
Author: ChangBo Guo(gcb) <email address hidden>
Date: Wed Apr 5 23:12:13 2017 +0800

    Fix test_request_too_large_no_content_length failure

    Webob >=1.7.0 change its way to check if body is readable [1],
    The is_body_readable() method returns False now. request.content_length
    will be set when set request.body automatically, and if
    request.context_length is None request.body will return returns b''.[3]
    So just set request.body in test.

    [1] https://github.com/Pylons/webob/commit/0e6fcbfbc58af6fdd4e125543bf8f5fb91b984c6
    [2] https://github.com/Pylons/webob/blame/master/webob/request.py#L699
    [3] https://github.com/Pylons/webob/blame/master/webob/request.py#L692

    Closes-Bug: #1679658

    Change-Id: I4e0932dd8664d96b2733c5d3b1fa846af0c7d132

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.middleware 3.26.0 release.