Zaza vault initialization often fails with CERTIFICATE_VERIFY_FAILED

[Aurelien Lourot]

Seen several times on OSCI, so far always on focal-ussuri and never on bionic-queens:

https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/761564
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/2/7686/index.html
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/2/7724/index.html

https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/770056
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/770056/1/7721/index.html

Traceback (most recent call last):
  File "/tmp/tmp.Ipg3xRHGbV/func/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 278, in main
    test_directory=args.test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 213, in func_test_runner
    force=force, test_directory=test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 143, in run_env_deployment
    test_directory=test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
    run_configure_list(functions)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/vault/setup.py", line 181, in auto_initialize
    validate_ca(cacertificate, application=validation_application)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/vault/setup.py", line 238, in validate_ca
    requests.get('https://{}:{}'.format(ip, str(port)), verify=fp.name)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='172.17.103.230', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

[Aurelien Lourot (aurelien-lourot)]

Actually this might have been fixed by recent changes in charmhelpers and charms.openstack. Re-triggering the linked reviews.

[Aurelien Lourot (aurelien-lourot)]

We'll have to get this one landed so that the suspected fixes make their way into the vault charm: https://review.opendev.org/c/openstack/charm-vault/+/770438

[Aurelien Lourot (aurelien-lourot)]

For the record, here are the fixes I had in mind: https://bugs.launchpad.net/charm-placement/+bug/1893847

There the bug has been marked as Invalid for the vault charm, so I'll go ahead and re-trigger the keystone-saml-mellon reviews now.

[Aurelien Lourot (aurelien-lourot)]

Happening again:
https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/761564
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/3/7824/index.html

We're getting a CERTIFICATE_VERIFY_FAILED in zaza/openstack/charm_tests/vault/setup.py:validate_ca(cacertificate, application=validation_application) with validation_application being keystone.

It seems indeed like the latest cert-related fixes may haven't made their way into the latest keystone charm yet as this review hasn't landed yet:
https://review.opendev.org/c/openstack/charm-keystone/+/771022