[Tempest] [SSL] 2 tests for object storage failed with ssl configuration

[Timur Nurlygayanov]

Steps To Reproduce:
1. Deploy environment with TLS
2. Run all Tempest tests for object storage using mos-tempest-runner scripts [1]

Expected Result:
All tests will pass

Observed Result:
2 tests will fail, if disable SSL, these tests pass.

The following tests failed:
test_web_listing_css
test_web_index

Tests failed with the following error:
traceback
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/tempest/tempest/test.py", line 171, in wrapper
    return func(*func_args, **func_kwargs)
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/object_storage/test_container_staticweb.py", line 134, in test_web_listing_css
    headers={})
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 627, in request
    headers=headers, body=body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 526, in _request
    req_url, method, headers=req_headers, body=req_body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 557, in raw_request
    headers=headers, body=body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/http.py", line 25, in request
    return super(ClosingHttp, self).request(*args, **new_kwargs)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1404, in _request
    redirections=redirections - 1)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/http.py", line 25, in request
    return super(ClosingHttp, self).request(*args, **new_kwargs)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python2.7/httplib.py", line 1073, in getresponse
    response.begin()
  File "/usr/local/lib/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.7/httplib.py", line 379, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

[1] https://github.com/Mirantis/mos-tempest-runner

[Timur Nurlygayanov (tnurlygayanov)]

Priority changed to High because two Tempest tests failed for any configurations with SSL and it looks like we don't have workaround for the issue.

[Timur Nurlygayanov (tnurlygayanov)]

Assigned to MOS Ceph team because we reproduced the issue but we don't know the root of the issue yet. It looks like we need someone with experience in object storage API and Swift.

[Roman Podoliaka (rpodolyaka)]

Need more time to understand the root cause. This may even be a problem of Tempest configuration. Moving to MU.

[Maksym Shalamov (mshalamov)]

Hello all,

Please let me to provide some explanation, related with the points listed in this bug report.

1) The following tempest tests for object storage testing failed in configuration with Swift:
             test_web_listing_css
             test_web_index
    Please find example of the trace here [1]
2) All tempest test for object storage testing failed in configuration with Ceph(RGW)
    Please find example of the trace here [2]

For more details please take a look into tempest reports:
1) swift https://drive.google.com/a/mirantis.com/file/d/0B68GLJA1XY9CMWFaaWFxVUpzWEE/view?usp=sharing
2) ceph https://drive.google.com/a/mirantis.com/file/d/0B68GLJA1XY9CNmx3SkJ3amxPd1E/view?usp=sharing

[1] http://paste.openstack.org/show/485936/
[2] http://paste.openstack.org/show/485939/

[Timur Nurlygayanov (tnurlygayanov)]

Any updates?

[Kostiantyn Danylov (kdanylov)]

You can override the 'shouldfail' tests in /home/developer/mos-tempest-runner/shouldfail/shouldfail.yaml
/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/__init__.py:28: DeprecationWarning: tempest-lib is deprecated for future bug-fixes and code changes in favor of tempest. Please change your imports from tempest_lib to tempest.lib
  DeprecationWarning)
Non-zero exit code (2) from test listing.
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
--- import errors ---
Failed to import test module: tempest.test_discover.test_discover
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/test_discover.py", line 18, in <module>
    from tempest.test_discover import plugins
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/plugins.py", line 20, in <module>
    from tempest_lib.common.utils import misc
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/utils/misc.py", line 19, in <module>
    from oslo_log import log as logging
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/log.py", line 49, in <module>
    from oslo_log import _options
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/_options.py", line 38, in <module>
    help='If set to true, the logging level will be set to '
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 965, in __init__
    super(BoolOpt, self).__init__(name, type=types.Boolean(), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mutable'

[Kostiantyn Danylov (kdanylov)]

I stop a test right before request using IPython and execute same request from console, using swift client:

swift --os-project-name tempest-StaticWebTest-1175394924 --os-username tempest-StaticWebTest-777055592 --os-password 'K8*KRpQh1jOyBVc' list tempest-TestContainer-951268031

this gives me expected object name "tempest-TestObject-284701657".

Same time request from tempest fail. Before fail it print a lot of warnings:

/home/rally/.rally/tempest/for-deployment-83d24637-32ce-4576-9ba8-e54ec2068072/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

which might be a reason for issue. In any case the reason is somewhere in tempest/rally.

[Oleksiy Butenko (obutenko)]

I check this issue on my env with ssl.
I stop a test before fail and execute request from console, using swift client and curl:
http://paste.openstack.org/show/508362/
So that the root cause - swift container doesn't available via https

on env without ssl curl return "test" - expected result for test

[Dina Belova (dbelova)]

Moving to 9.0-updates after communication with Konstantin. It's too late trying fix it in 9.0

[Pete Zaitcev (zaitcev)]

I looked into a situation where these two tests, test_web_index and test_web_listing_css, fail when running through HAproxy that terminates SSL. Here's what I found.

At a certain point, Tempest does GET on /v1/AUTH_fa4d5382f43b41d59f0fb51ef330077b/tempest-TestContainer-2075955620 without a trailing slash. The staticweb returns a 301 redirect with Location: http://10.0.0.101:13808/v1/AUTH_fa4d5382f43b41d59f0fb51ef330077b/tempest-TestContainer-2075955620/

Basically all it wants is to add the trailing slash. It does so by returning HTTPMovedPermanently(location=env['PATH_INFO']+'/'). The path_info at this point is just a path. Before the Response.__call__() returns, swob prepends the netloc to the location, and mistakenly uses the http: scheme.

The urllib3 then uses the Location from the redirect and attempts to talk HTTP to HAproxy that is listening with SSL. HApxoxy resets the connectin. The process repeats until urllib3 connection exhausts its retries.

The right fix needs the scheme to be correct in responses. It can either be set by staticweb (tempauth does that), or extracted from wsgi.url_scheme (which requires magic and may be impossible).

[Pete Zaitcev (zaitcev)]

I discussed this on #openstack-swift and surprisingly, Clay et.al. were in favour of additional configuration. I came around to this approach too, when I found that my stunnel does not provide any kind of header at all and thus the automatic guessing is impossible.

Here's a proposed patch:
 https://review.openstack.org/372809

Note that this still needs OOO or other orchestration to set url_base. With the patch above, one can set url_base to "https://", in which case hostname is taken from Host: header as usual.

[Sergey Shevorakov (sshevorakov)]

ETA: 08/03/2016

[Kostiantyn Danylov (kdanylov)]

AFAIK this bug doesn't feet into MU requirements. We have never see it in production, the only failing code, is a unit-test. So I put 'wont-fix' for 8.0 and 7.0

[Vitaly Sedelnik (vsedelnik)]

Fix Committed for 9.2, the fix is https://review.fuel-infra.org/#/c/29499/

[Oleksiy Butenko (obutenko)]

if you need access to env and steps to reproduce - contact me
reproduced again on MOS 9.2 snapshot #744

[Vitaly Sedelnik (vsedelnik)]

More time is needed to get this issue fixed - retargeted to 9.3

[Alexander Rubtsov (arubtsov)]

sla1 for 9.0-updates

[Oleksiy Butenko (obutenko)]

we need configure base_url in /etc/swift/proxy-server.conf for mos 10.
If 9.x release (stable\mitaka) had all releated patches - need create the same patch(If not - create backports before)

[Oleksiy Butenko (obutenko)]

I can't reproduce this error on latest 9.x deploy.(test failed with another error)
After debug issue we will prepare fix for 9.2

[Nikita Karpin (mkarpin)]

fixes for master, ocata, newton - https://review.openstack.org/#/q/topic:bug/1537071

[Oleksiy Butenko (obutenko)]

https://review.openstack.org/#/c/442600/ - fix for mitaka

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 + mu1 updates.

Environment:
3 controllers + 1 compute node with enabled TLS/HTTPS.

Actual results:
The following tempest tests are passed after the fix (http://paste.openstack.org/show/602792/):
test_web_index
test_web_listing_css

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library ocata-eol release.