Tempest test test_get_version_details is failed

[Sofiia Andriichenko]

Detailed bug description:
tempest.scenario.test_object_storage_telemetry_middleware.TestObjectStorageTelemetry

Configuration:
    ISO: http://paste.openstack.org/show/495771/
  Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - LVM
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, ironic, cinder, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.api.compute.test_versions.TestVersions.test_get_version_details

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)
snapshot https://drive.google.com/file/d/0B8hkiEm94sEtSmM1emJiZGlXSms/view?usp=sharing

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-6cf6f710-6362-4d3a-a8f6-86b8bd5dd1dd/tempest/api/compute/test_versions.py", line 68, in test_get_version_details
    result = self.versions_client.get_version_by_url(link['href'])
  File "/home/rally/.rally/tempest/for-deployment-6cf6f710-6362-4d3a-a8f6-86b8bd5dd1dd/tempest/lib/services/compute/versions_client.py", line 58, in get_version_by_url
    {'X-Auth-Token': self.token})
  File "/home/rally/.rally/tempest/for-deployment-6cf6f710-6362-4d3a-a8f6-86b8bd5dd1dd/tempest/lib/common/rest_client.py", line 559, in raw_request
    headers=headers, body=body)
  File "/home/rally/.rally/tempest/for-deployment-6cf6f710-6362-4d3a-a8f6-86b8bd5dd1dd/tempest/lib/common/http.py", line 54, in request
    *args, **new_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 69, in request
    **urlopen_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 90, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 165, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 623, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/retry.py", line 281, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=u'public.fuel.local', port=8774): Max retries exceeded with url: /v2/ (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",)))

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced on customer environment with MOS 8.0, baremetal lab with 10+ hardware servers, Ceph+RadosGW backend, HA.

[Roman Podoliaka (rpodolyaka)]

I believe this is the same as https://bugs.launchpad.net/glance/+bug/1384379 . We need to make we either use a helper oslo.middleware tools in api-paste.ini or configure this config option in Nova - https://review.openstack.org/#/c/206479/2/nova/wsgi.py

[Oleksiy Butenko (obutenko)]

contact me for access to the environment where tests are failing.

[Pavel Kholkin (pkholkin)]

As Roman pointed above, changing Nova config option helps with this problem. The traceback shows that nova-api couldn't be reached via http (but it could be reached via https). Adding 'secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO' opt to [DEFAULT] group to nova.conf file solves the issue.

Seems like this problem should be fixed from the Puppet side. Moreover it should be fixed in general for other components too (NOT only for Nova). Maybe there were no similar tests in other components causing to the same problem.

[Roman Podoliaka (rpodolyaka)]

^ the only concern I have is that there might be no such option in other components. But we could probably consider giving oslo.middleware a try in api-paste.ini's.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: mos-8.0
Change author: Mykyta Karpin <email address hidden>
Review: https://review.fuel-infra.org/20850

[Ivan Berezovskiy (iberezovskiy)]

https://review.openstack.org/#/c/318157/ - puppet nova patch, after merge it'll be backported to stable/mitaka. For mos-8.0 - https://review.fuel-infra.org/20850

[Timur Nurlygayanov (tnurlygayanov)]

Can we prepare the backport for 9.0 before HCF?

[Timur Nurlygayanov (tnurlygayanov)]

Marked as Confirmed for 9.0 because patch is not ready for 9.0

[Nikita Karpin (mkarpin)]

https://review.openstack.org/#/c/319167 - backport to stable/mitaka

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: mos-8.0
Change author: Mykyta Karpin <email address hidden>
Review: https://review.fuel-infra.org/21135

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mykyta Karpin <email address hidden> on branch: mos-8.0
Review: https://review.fuel-infra.org/20850
Reason: in favor of https://review.fuel-infra.org/#/c/21135/

[Ivan Berezovskiy (iberezovskiy)]

Fuel patch for master - https://review.openstack.org/#/c/318602/, for mitaka - https://review.openstack.org/#/c/320342/

[Sergii Turivnyi (sturivnyi)]

Can reproduce on
9.0 mos iso #427
 Traceback (most recent call last):

  File "/home/rally/.rally/tempest/for-deployment-3863ad5b-5540-4e87-914b-e5a6d773a227/tempest/api/compute/test_versions.py", line 68, in test_get_version_details

    result = self.versions_client.get_version_by_url(link['href'])

  File "/home/rally/.rally/tempest/for-deployment-3863ad5b-5540-4e87-914b-e5a6d773a227/tempest/lib/services/compute/versions_client.py", line 58, in get_version_by_url

    {'X-Auth-Token': self.token})

  File "/home/rally/.rally/tempest/for-deployment-3863ad5b-5540-4e87-914b-e5a6d773a227/tempest/lib/common/rest_client.py", line 559, in raw_request

    headers=headers, body=body)

  File "/home/rally/.rally/tempest/for-deployment-3863ad5b-5540-4e87-914b-e5a6d773a227/tempest/lib/common/http.py", line 54, in request

    *args, **new_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 69, in request

    **urlopen_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 90, in request_encode_url

    return self.urlopen(method, url, **extra_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 165, in urlopen

    response = conn.urlopen(method, u.request_uri, **kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 651, in urlopen

    release_conn=release_conn, **response_kw)

  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 623, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/retry.py", line 281, in increment

    raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=u'public.fuel.local', port=8774): Max retries exceeded with url: /v2/ (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",)))

[Nikita Karpin (mkarpin)]

it is because of incorrect section, related bug - https://bugs.launchpad.net/puppet-nova/+bug/1588360,
fix for it is https://review.openstack.org/#/c/324530/

[Ivan Berezovskiy (iberezovskiy)]

Patch is merged - https://review.openstack.org/#/c/324530/

[Oleksiy Butenko (obutenko)]

verified on 9.0 mos iso #452

[Anna Babich (ababich)]

It passes on Devstack with Newton:
oot@ubuntu:~# rally verify start --regex tempest.api.compute.test_versions.TestVersions.test_get_version_details

{0} tempest.api.compute.test_versions.TestVersions.test_get_version_details [1.168156s] ... ok

======
Totals
======
Ran: 1 tests in 3.0000 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1.1682 sec.