[bionic][ussuri] functest failing due to connection reset during setup

[Alex Kavanagh]

The section it got to:

2020-10-05 13:38:37 [INFO] Using keystone API V3 (or later) for overcloud auth
2020-10-05 13:38:43 [INFO] Downloading image bionic
2020-10-05 13:40:10 [INFO] active
2020-10-05 13:40:10 [INFO] Setting image properties: None
2020-10-05 13:41:34 [INFO] Using keystone API V3 (or later) for overcloud auth
2020-10-05 13:42:39 [ERROR] {'default_alias': 'zaza-ab0a67dae570'}
2020-10-05 13:42:39 [ERROR] Model default_alias (zaza-ab0a67dae570)

i.e. it didn't finish setting up.

The model looks stable, but it could be due to a hook still running (i.e. the check for stability wasn't true).

The traceback:

Traceback (most recent call last):
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 136, in run_env_deployment
    config_steps.get(deployment.model_alias, []))
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
    run_configure_list(functions)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/keystone/setup.py", line 143, in add_demo_user
    _v3()
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/keystone/setup.py", line 76, in _v3
    enabled=True)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneclient/v3/domains.py", line 58, in create
    **kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneclient/base.py", line 86, in func
    return f(*args, **new_kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneclient/base.py", line 386, in create
    self.key)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneclient/base.py", line 192, in _post
    resp, body = self.client.post(url, body=body, **kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 401, in post
    return self.request(url, 'POST', **kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 554, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 257, in request
    return self.session.request(url, method, **kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneauth1/session.py", line 921, in request
    resp = send(**kwargs)
  File "/tmp/tmp.IOhetQeOuh/func/lib/python3.5/site-packages/keystoneauth1/session.py", line 1028, in _send_request
    raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.17.112.230:35357/v3/domains: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

[David Ames (thedac)]

"('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')" is almost certainly caused by haproxy timing out and dropping the connection.

It is likely due to a busy undercloud causing connections to pile up on keystone units and eventually timeout. This is difficult to prove without having a process list and cpu usage metrics.

Potential path forward:

Since keystone is central to the entire cloud we should give it the resources to handle the barrage of requests.

* Bump the constraints for keystone units so they have more RAM
* Up the number of threads keystone runs with worker-multiplier (if necessary, the bump may do this for us)

Note: we have seen similar with neutron-api and other API charms. It is possible we should do this for all API charms.