Containerized gate is broken due to OpenStack API certs missing

[Michal Dulko]

Our containerized gates are broken at the moment, because of healthchecks being unable to connect to Keystone API due to certificates failure. Exact cause is still unknown.

Mar 22 11:50:32.968611 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health [-] Error when creating a Keystone client: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/stack/data/ca-bundle.pem.: IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/stack/data/ca-bundle.pem
Mar 22 11:50:32.968857 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health Traceback (most recent call last):
Mar 22 11:50:32.969100 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/managers/health.py", line 74, in readiness_status
Mar 22 11:50:32.969336 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health self.verify_keystone_connection()
Mar 22 11:50:32.969578 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/managers/health.py", line 124, in verify_keystone_connection
Mar 22 11:50:32.969796 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health endpoint_type=endpoint_type)
Mar 22 11:50:32.970098 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneclient/client.py", line 62, in Client
Mar 22 11:50:32.970313 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health d = discover.Discover(session=session, **kwargs)
Mar 22 11:50:32.970530 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneclient/discover.py", line 178, in __init__
Mar 22 11:50:32.970744 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health authenticated=authenticated)
Mar 22 11:50:32.970965 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line 143, in __init__
Mar 22 11:50:32.971207 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health authenticated=authenticated)
Mar 22 11:50:32.971422 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneclient/_discover.py", line 38, in get_version_data
Mar 22 11:50:32.971655 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health resp = session.get(url, headers=headers, authenticated=authenticated)
Mar 22 11:50:32.971895 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get
Mar 22 11:50:32.972108 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health return self.request(url, 'GET', **kwargs)
Mar 22 11:50:32.972337 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request
Mar 22 11:50:32.972551 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health auth_headers = self.get_auth_headers(auth)
Mar 22 11:50:32.972769 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers
Mar 22 11:50:32.972982 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health return auth.get_headers(self, **kwargs)
Mar 22 11:50:32.973194 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
Mar 22 11:50:32.973438 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health token = self.get_token(session)
Mar 22 11:50:32.973661 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
Mar 22 11:50:32.973872 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health return self.get_access(session).auth_token
Mar 22 11:50:32.974094 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
Mar 22 11:50:32.974325 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health self.auth_ref = self.get_auth_ref(session)
Mar 22 11:50:32.974546 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
Mar 22 11:50:32.974758 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health self._plugin = self._do_create_plugin(session)
Mar 22 11:50:32.974980 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
Mar 22 11:50:32.975197 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health authenticated=False)
Mar 22 11:50:32.975416 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 556, in get_discovery
Mar 22 11:50:32.975628 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health authenticated=authenticated)
Mar 22 11:50:32.975837 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 1171, in get_discovery
Mar 22 11:50:32.976048 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health disc = Discover(session, url, authenticated=authenticated)
Mar 22 11:50:32.976261 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 401, in __init__
Mar 22 11:50:32.976479 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health authenticated=authenticated)
Mar 22 11:50:32.976689 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 99, in get_version_data
Mar 22 11:50:32.976911 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health resp = session.get(url, headers=headers, authenticated=authenticated)
Mar 22 11:50:32.977147 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get
Mar 22 11:50:32.977361 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health return self.request(url, 'GET', **kwargs)
Mar 22 11:50:32.977725 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 698, in request
Mar 22 11:50:32.978053 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health resp = send(**kwargs)
Mar 22 11:50:32.978406 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 756, in _send_request
Mar 22 11:50:32.978752 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health resp = self.session.request(method, url, **kwargs)
Mar 22 11:50:32.979111 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
Mar 22 11:50:32.979426 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health resp = self.send(prep, **send_kwargs)
Mar 22 11:50:32.979644 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
Mar 22 11:50:32.979868 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health r = adapter.send(request, **kwargs)
Mar 22 11:50:32.980096 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
Mar 22 11:50:32.980328 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health self.cert_verify(conn, request.url, verify, cert)
Mar 22 11:50:32.980548 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 226, in cert_verify
Mar 22 11:50:32.980782 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health "invalid path: {0}".format(cert_loc))
Mar 22 11:50:32.981299 ubuntu-xenial-rax-dfw-0003126095 kubectl[32108]: 2018-03-22 11:50:32.965 1 ERROR kuryr_kubernetes.controller.managers.health IOError: Could not find a suitable TLS CA certificate bundle, invalid path:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/555502

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555502
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=3b7e518a940cffbaab1aa52f5a3166175ee3dd6c
Submitter: Zuul
Branch: master

commit 3b7e518a940cffbaab1aa52f5a3166175ee3dd6c
Author: Michał Dulko <email address hidden>
Date: Thu Mar 22 18:28:28 2018 +0100

    Add CA certificates Secret and mount it

    Our containerized gates started failing recently. Turns out some default
    configuation was changed and `tls-proxy` service was added. This option
    makes all OpenStack endpoints use HTTPS. This includes creation of a
    DevStack CA certificates bundle that then will be configured to be
    verified when connecting to OpenStack APIs. This works well with
    non-containerized deployment as the bundle is available locally in
    /opt/stack/data and our `[neutron]` section sets `cafile` option to
    point there.

    Things are different in containerized deployment use case as we need a
    way to pass those certificates into the container. Effectively - we had
    no CA certificates support for containerized deployments either in
    DevStack or production.

    This commit adds that support by including new Kuryr Kubernetes resource
    definition - `kuryr-certificates` Secret. It is supposed to hold CA
    certificate under `kuryr-ca-bundle.crt` key. kuryr-controller DaemonSet
    definition was modified to mount the certificate into /etc/ssl/certs.

    Changes also include implementing support for that in DevStack plugin
    (placing the certificate in the secret and setting the `[neutron]cafile`
    config option to point to that certificate).

    Closes-Bug: 1758061
    Change-Id: I7ac9d05868994cfc2a1aef4a8cd6c2148895e9c8

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-kubernetes 0.5.0 release.