Rally scenarios for heat failed by 'Connection aborted'

[Sergey Galkin]

api: '1.0'
astute_sha: cff84956fbe8ac8f95020aa04d326230cae92158
auth_required: true
build_id: 2014-11-06_21-29-20
build_number: '82'
feature_groups:
- mirantis
fuellib_sha: ef9f6407d0d555346ac34b9f29ee004083f8e6d2
fuelmain_sha: 0d3f16254760999f90564c384f34fda8d6c82f51
nailgun_sha: 172c1acadabf59b9434718d9a1c27a3a4db5f3cc
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'

Steps to reproduce
1. Deploy 100 nodes cluster with 3 controllers in HA and 97 computes with Cinder with ISCSI with neutron GRE
2. Run rally tests

config
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 5,
      "times": 97
    },
    "context": {
      "users": {
        "project_domain": "default",
        "concurrent": 30,
        "user_domain": "default",
        "tenants": 1,
        "users_per_tenant": 1
      }
    }
  },
  "name": "HeatStacks.create_and_list_stack",
  "pos": 0
}

All tests failed with error ('Connection aborted.', BadStatusLine("''",))
results
+--------+--------+--------+--------+----------+-------+--------+
| Action | min | avg | max | failures | total | result |
+--------+--------+--------+--------+----------+-------+--------+
| ~total | 13.171 | 13.511 | 15.253 | 97 | 97 | :( |
+--------+--------+--------+--------+----------+-------+--------+
stderr-0000
<class 'requests.exceptions.ConnectionError'>
('Connection aborted.', BadStatusLine("''",))
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/stacks.py", line 55, in create_and_list_stack
    self._create_stack(stack_name, template)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 256, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/utils.py", line 90, in _create_stack
    stack = self.clients("heat").stacks.get(stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 146, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 254, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 187, in _http_request
    **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))
stderr-0001
<class 'requests.exceptions.ConnectionError'>
('Connection aborted.', BadStatusLine("''",))
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/stacks.py", line 55, in create_and_list_stack
    self._create_stack(stack_name, template)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 256, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/utils.py", line 89, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 119, in create
    data=kwargs, headers=headers)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 254, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 187, in _http_request
    **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))

or run rally tests
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 5,
      "times": 97
    },
    "args": {
      "template_path": "/opt/stack/rally-scenarios/heat/server_with_ports.yml"
    },
    "context": {
      "users": {
        "project_domain": "default",
        "concurrent": 30,
        "user_domain": "default",
        "tenants": 2,
        "users_per_tenant": 3
      }
    }
  },
  "name": "HeatStacks.create_and_delete_stack",
  "pos": 0
}

All tests failed with error "HTTPException: ERROR: <html><body><h1>504 Gateway Time-out</h1>"

results
+--------+--------+--------+--------+----------+-------+--------+
| Action | min | avg | max | failures | total | result |
+--------+--------+--------+--------+----------+-------+--------+
| ~total | 60.131 | 60.391 | 61.404 | 97 | 97 | :( |
+--------+--------+--------+--------+----------+-------+--------+
stderr-0000
<class 'heatclient.exc.HTTPException'>
ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/stacks.py", line 74, in create_and_delete_stack
    stack = self._create_stack(stack_name, template)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 256, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/utils.py", line 90, in _create_stack
    stack = self.clients("heat").stacks.get(stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 146, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 254, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 209, in _http_request
    raise exc.from_response(resp)
HTTPException: ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

stderr-0001
<class 'heatclient.exc.HTTPException'>
ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/stacks.py", line 74, in create_and_delete_stack
    stack = self._create_stack(stack_name, template)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 256, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/heat/utils.py", line 90, in _create_stack
    stack = self.clients("heat").stacks.get(stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 146, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 254, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 209, in _http_request
    raise exc.from_response(resp)
HTTPException: ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

I can't attach snapshot because
du -sh /var/log/
22G /var/log/

[Pavlo Shchelokovskyy (pshchelo)]

I am more than sure that is the problem of our current heat-engine deployment - there is only one of them running even in HA deployment, and it might simply choke on load, as the actual job of creating and deleting stacks is done by this single heat-engine.

This behavior of heat-engine was a workaround for (then undiscovered) bug https://bugs.launchpad.net/mos/+bug/1387345 that prevented us from actually using Heat in multi-engine mode.

Patches with the fix that will allow us to really use active-active heat-engine deployment are
- https://review.openstack.org/#/c/133455/ (fuel-web/master, merged)
- https://review.openstack.org/#/c/133263/ (fuel-lib/master, on review)

[Sergey Kraynev (skraynev)]

Second patch was merged (https://review.openstack.org/#/c/133263/)

Need re-check current issue.

[Pavlo Shchelokovskyy (pshchelo)]

I see those patches landed in fuel master, but have they landed in 6.0 branches? Are they in latest 6.0 iso being tested in scale? As those errors do still appear there.