[Tempest] [SSL] Heat tests failed with SSL deployments

[Timur Nurlygayanov]

Note:
This issue reproduced on MOS 7.0 and MOS 8.0.

Steps To Reproduce:
1. Deploy environment with TLS
2. Run all Tempest tests for Heat using mos-tempest-runner scripts [1]

Expected Result:
All tests will pass

Observed Result:
During creation a stack test tries to get keystone trust through HTTP protocol, but SSL is enabled, so it is possible to get trust only with HTTPS. If disable SSL, these tests pass.

Some tests with deployment of Heat stacks failed with the following errors:

traceback
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/tempest/tempest/test.py", line 272, in setUpClass
    six.reraise(etype, value, trace)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test.py", line 265, in setUpClass
    cls.resource_setup()
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/orchestration/stacks/test_neutron_resources.py", line 73, in resource_setup
    'SubNetCidr': str(cls.subnet_cidr)
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/orchestration/base.py", line 75, in create_stack
    files=files)
  File "/home/developer/mos-tempest-runner/tempest/tempest/services/orchestration/json/orchestration_client.py", line 56, in create_stack
    resp, body = self.post(uri, headers=headers, body=body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 640, in request
    resp, resp_body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 759, in _error_checker
    message=message)
tempest_lib.exceptions.ServerFault: Got server fault
Details: Remote error: ConnectionRefused Unable to establish connection to http://10.109.56.3:35357/v3/OS-TRUST/trusts

[1] https://github.com/Mirantis/mos-tempest-runner

[Timur Nurlygayanov (tnurlygayanov)]

Status changed to Critical because it will affect all deployments with Heat + SSL endpoints.

[Timur Nurlygayanov (tnurlygayanov)]

Comments from Alexey Odinokov:

Here is a workaround for the 1st issue, that Sergey K helped to find.
You may want to add this to the corresponded issue ticket :)

Heat need some modification to be able to work with HTTPS public admin endpoint
on each controller node:
#vim /etc/heat/heat.conf
comment
 #auth_host = 192.168.0.2
 #auth_port = 35357
 #auth_protocol = http
and add
identity_uri = https://kiloapi.ixcloud.net:35357/

#vim /etc/keystone/keystone.conf
add
admin_endpoint = https://kiloapi.ixcloud.net:35357

restart services
crm resource restart clone_p_heat-engine
service keystone restart
service apache2 restart

[Timur Nurlygayanov (tnurlygayanov)]

So, we have the workaround, priority changed to High then.

[Pavlo Shchelokovskyy (pshchelo)]

As this is a matter of how we configure Heat, shouldn't this be assigned to fuel-library team?

[Timur Nurlygayanov (tnurlygayanov)]

MOS Puppet team, could you please take a look and fix the issue? (need to fix the configuration for setups with ssl endpoints)

[Ivan Berezovskiy (iberezovskiy)]

Timur, please try to reproduce this issue on Fuel 8.0 ISO

[Timur Nurlygayanov (tnurlygayanov)]

Ok, we are going to check in on the latest MOS 8.0 builds.

[Oleksiy Butenko (obutenko)]

tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_network[id-c572b915-edb1-4e90-b196-c7199a6848c0,network] Failed
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_resources[id-f9e2664c-bc44-4eef-98b6-495e4f9d74b3] Failed
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_router[id-96af4c7f-5069-44bc-bdcf-c0390f8a67d1,network] Failed
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_server[compute,id-75d85316-4ac2-4c0e-a1a9-edd2148fc10e,network] Failed
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_router_interface[id-89f605bd-153e-43ee-a0ed-9919b63423c5,network] Failed
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_subnet[id-e8f84b96-f9d7-4684-ad5f-340203e9f2c2,network] Failed

Traceback (most recent call last):
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/tempest/tempest/test.py", line 272, in setUpClass
    six.reraise(etype, value, trace)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test.py", line 265, in setUpClass
    cls.resource_setup()
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/orchestration/stacks/test_neutron_resources.py", line 87, in resource_setup
    server_id = body['physical_resource_id']
KeyError: 'physical_resource_id'

[Oleksiy Butenko (obutenko)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Timur Nurlygayanov (tnurlygayanov)]

We know how to fix the issue (please see my comment #2) but we need to update Heat configuration file for this.
Heat team, please review the configuration and confirm that it will be correct fix for the issue.

Then please assign to MOS puppet team who should fix the issue.

[Sergey Kraynev (skraynev)]

Timur: I confirm, that this fix worked for deployments, where this issues was discovered. So I will re-assigned this bug on Puppet Team. P.s. afaik, they have a patch on review (don't know, why it was no mentioned here automatically.)

[Roman Podoliaka (rpodolyaka)]

I'd really like us to avoid trying things at random here: from what is stated in comment #2 you propose to change the management endpoint of Keystone from HTTP to HTTPS, which *must not* be the case for the SSL configuration we have in MOS 8.0, which only applies to public endpoints.

At this point, this seems to be a purely Tempest problem to me. Heat and Keystone are configured and work as expected.

[Roman Podoliaka (rpodolyaka)]

Timur, we need expertise of your team here ^

[Timur Nurlygayanov (tnurlygayanov)]

It looks like the root of the issue in mos-tempest-runner scripts/Tempest itself, and Heat was deployed correctly "from the box".

[Roman Podoliaka (rpodolyaka)]

So, per conversation with Timur and Sergey and Slack:

this seems to be caused by mos-tempest runner, which changes the default Keystone endpoints and makes admin endpoints public

I suggest we mark this as "non-release" (no changes must be done to release artifacts - packages or Puppet manifests), leave this on MOS-QA and let them fix mos-tempest runner

[Timur Nurlygayanov (tnurlygayanov)]

The reason of this fail in incorrect test scripts which changed public endpoint of cluster to run Tempest tests.
We need to avoid such "effects" in test scripts, this is why we are going to use Rally verify instead of mos-tempest-runner scripts.

Bug closed as Won't Fix because it doesn't affect MOS releases, is affects only our test scripts which run Tempest.