policy.d test is unstable

[Alex Kavanagh]

From: https://review.opendev.org/#/c/732673/
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-openstack-dashboard/732673/2/5962/test_charm_func_full_9103/func.txt

ERROR
2020-06-05 11:10:32 [INFO] ======================================================================
2020-06-05 11:10:32 [INFO] ERROR: test_003_test_overide_is_observed (zaza.openstack.charm_tests.openstack_dashboard.tests.OpenStackDashboardPolicydTests)
2020-06-05 11:10:32 [INFO] Test that the override is observed by the underlying service.
2020-06-05 11:10:32 [INFO] ----------------------------------------------------------------------
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 11:10:32 [INFO] chunked=chunked,
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 11:10:32 [INFO] six.raise_from(e, None)
2020-06-05 11:10:32 [INFO] File "<string>", line 3, in raise_from
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 11:10:32 [INFO] httplib_response = conn.getresponse()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 11:10:32 [INFO] response.begin()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 11:10:32 [INFO] version, status, reason = self._read_status()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 11:10:32 [INFO] raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 11:10:32 [INFO] http.client.RemoteDisconnected: Remote end closed connection without response
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
2020-06-05 11:10:32 [INFO] timeout=timeout
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
2020-06-05 11:10:32 [INFO] method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/util/retry.py", line 403, in increment
2020-06-05 11:10:32 [INFO] raise six.reraise(type(error), error, _stacktrace)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/packages/six.py", line 734, in reraise
2020-06-05 11:10:32 [INFO] raise value.with_traceback(tb)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 11:10:32 [INFO] chunked=chunked,
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 11:10:32 [INFO] six.raise_from(e, None)
2020-06-05 11:10:32 [INFO] File "<string>", line 3, in raise_from
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 11:10:32 [INFO] httplib_response = conn.getresponse()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 11:10:32 [INFO] response.begin()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 11:10:32 [INFO] version, status, reason = self._read_status()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 11:10:32 [INFO] raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 11:10:32 [INFO] urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/policyd/tests.py", line 475, in test_003_test_overide_is_observed
2020-06-05 11:10:32 [INFO] self.get_client_and_attempt_operation(self.keystone_ips[0])
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/openstack_dashboard/tests.py", line 471, in get_client_and_attempt_operation
2020-06-05 11:10:32 [INFO] result = client.get(_url)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 543, in get
2020-06-05 11:10:32 [INFO] return self.request('GET', url, **kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
2020-06-05 11:10:32 [INFO] resp = self.send(prep, **send_kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 665, in send
2020-06-05 11:10:32 [INFO] history = [resp for resp in gen] if allow_redirects else []
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 665, in <listcomp>
2020-06-05 11:10:32 [INFO] history = [resp for resp in gen] if allow_redirects else []
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 245, in resolve_redirects
2020-06-05 11:10:32 [INFO] **adapter_kwargs
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
2020-06-05 11:10:32 [INFO] r = adapter.send(request, **kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/adapters.py", line 498, in send
2020-06-05 11:10:32 [INFO] raise ConnectionError(err, request=request)
2020-06-05 11:10:32 [INFO] requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/policyd/tests.py", line 480, in test_003_test_overide_is_observed
2020-06-05 11:10:32 [INFO] .format(str(e)))
2020-06-05 11:10:32 [INFO] zaza.openstack.utilities.exceptions.PolicydError: Service action failed and should have passed after removing policy override: "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))"

This probably means that the apache webserver wasn't back in time; perhaps some retries to allow it to come back up with help stabilise the test.

[Aurelien Lourot (aurelien-lourot)]

Looking at the logs, I don't believe the webserver wasn't back and ready. Also the code that failed is:

def _login(dashboard_ip, domain, username, password):
    # [...]
    auth_url = 'http://{}/horizon/auth/login/'.format(dashboard_ip)

    # start session, get csrftoken
    client = requests.session()
    client.get(auth_url)

def get_client_and_attempt_operation(self, ip):
    # [...]
    client, response = _login(
        unit.public_address, domain, username, password)
    # now attempt to get the domains page
    _url = self.url.format(unit.public_address)
    result = client.get(_url)

_login() (and its client.get()) succeeded (so the webserver was ready) whereas the later client.get() failed.