Opening TLS causes SSLError

Bug #1873736 reported by Zijian Guo on 2020-04-20
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
masakari
Medium
Mark Goddard
Stein
Medium
Radosław Piliszek
Train
Medium
Unassigned
Ussuri
Medium
Unassigned
Victoria
Medium
Mark Goddard
masakari-dashboard
Medium
Mark Goddard
Stein
Medium
Unassigned
Train
Medium
Unassigned
Ussuri
Medium
Unassigned
Victoria
Medium
Mark Goddard
masakari-monitors
Medium
Mark Goddard
Stein
Medium
Unassigned
Train
Medium
Unassigned
Ussuri
Medium
Unassigned
Victoria
Medium
Mark Goddard

Bug Description

No certificate was passed in keystone session.

2020-04-20 14:13:24.320 44 INFO masakari.compute.nova [req-900b346e-0441-4f71-86ea-b2357248a389 nova - - - -] Call hypervisor search command to get list of matching hypervisor name 'compute1'
2020-04-20 14:13:24.347 44 WARNING keystoneauth.identity.generic.base [req-900b346e-0441-4f71-86ea-b2357248a389 nova - - - -] Failed to discover available identity versions when contacting https://vip.qs.in:5000/v3. Attempting to parse version from URL.: SSLError: SSL exception connecting to https://vip.qs.in:5000/v3: HTTPSConnectionPool(host='vip.qs.in', port=5000): Max retries exceeded with url: /v3 (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions [req-900b346e-0441-4f71-86ea-b2357248a389 nova - - - -] Unexpected exception in API method: MasakariException: An unknown exception occurred.
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions Traceback (most recent call last):
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/api/openstack/extensions.py", line 389, in wrapped
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return f(*args, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/api/validation/__init__.py", line 56, in wrapper
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return func(*args, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/api/openstack/ha/hosts.py", line 109, in create
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions host = self.api.create_host(context, segment_id, host_data)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/ha/api.py", line 200, in create_host
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions self._is_valid_host_name(context, host.name)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/ha/api.py", line 154, in _is_valid_host_name
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions novaclient.hypervisor_search(context, name)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/compute/nova.py", line 63, in wrapper
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions _reraise(exception.MasakariException(reason=err_msg))
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/compute/nova.py", line 50, in _reraise
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/compute/nova.py", line 58, in wrapper
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions res = method(self, ctx, *args, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/masakari/compute/nova.py", line 254, in hypervisor_search
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions hypervisors_list = nova.hypervisors.search(hypervisor_name)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/novaclient/v2/hypervisors.py", line 111, in search
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self._list(url, 'hypervisors')
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions resp, body = self.api.client.get(url)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self.request(url, 'GET', **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 72, in request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self.session.request(url, method, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 747, in request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions auth_headers = self.get_auth_headers(auth)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1158, in get_auth_headers
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return auth.get_headers(self, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions token = self.get_token(session)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self.get_access(session).auth_token
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions self.auth_ref = self.get_auth_ref(session)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions authenticated=False, log=False, **rkwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1106, in post
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions return self.request(url, 'POST', **kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 888, in request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions resp = send(**kwargs)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 983, in _send_request
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions raise exceptions.SSLError(msg)
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions MasakariException: An unknown exception occurred.
2020-04-20 14:13:24.364 44 ERROR masakari.api.openstack.extensions

Fix proposed to branch: master
Review: https://review.opendev.org/721174

Changed in masakari:
assignee: nobody → Zijian Guo (zijianguo)
status: New → In Progress
Radosław Piliszek (yoctozepto) wrote :

This applies to more than Masakari engine. It seems Masakari dashboard and monitors misbehave the same (regarding Masakari itself).

Changed in masakari-monitors:
status: New → Confirmed
Radosław Piliszek (yoctozepto) wrote :

It seems there is no launchpad project for masakari-dashboard.

Fix proposed to branch: master
Review: https://review.opendev.org/742634

Changed in masakari:
assignee: Zijian Guo (zijianguo) → Mark Goddard (mgoddard)

Fix proposed to branch: master
Review: https://review.opendev.org/742637

Changed in masakari-monitors:
assignee: nobody → Mark Goddard (mgoddard)
status: Confirmed → In Progress

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: master
Review: https://review.opendev.org/721174
Reason: merging the other patch

Changed in masakari:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/742634
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=3bd82c776c91bdd1caee474f3fc9cda18927da7f
Submitter: Zuul
Branch: master

commit 3bd82c776c91bdd1caee474f3fc9cda18927da7f
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 18:57:21 2020 +0000

    Fix CA file for nova client

    If a custom CA file is configured via nova_ca_certificates_file,
    currently communication with Keystone will fail, since the session is
    not created using this CA file. The same is true for nova_api_insecure.

    This change fixes the issue by using a keystoneauth session loader.

    Closes-Bug: #1873736

    Change-Id: I54a4f398a06c61e0f6f8f3efd3e4b6214bca756b

Reviewed: https://review.opendev.org/742637
Committed: https://git.openstack.org/cgit/openstack/masakari-monitors/commit/?id=e70404588002222c8475667ff1a94f4c265a8def
Submitter: Zuul
Branch: master

commit e70404588002222c8475667ff1a94f4c265a8def
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 19:02:28 2020 +0000

    Use keystoneauth1 config option loading for masakari client

    If a custom CA file is configured via [api] cafile, currently
    communication with Keystone will fail, since the session is not created
    using this CA file. The [api] insecure option is also ignored.

    This change fixes the issue by using keystoneauth loading for the auth
    and session, to ensure all standard configuration options are supported.

    Change-Id: Idd58b72f7f5242e8135fec71b42adf5dd1852417
    Closes-Bug: #1873736

Changed in masakari-monitors:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/749965
Committed: https://git.openstack.org/cgit/openstack/masakari-dashboard/commit/?id=59ee1d59a1e08cd2245a8e876ea8931419ac5e32
Submitter: Zuul
Branch: master

commit 59ee1d59a1e08cd2245a8e876ea8931419ac5e32
Author: Mark Goddard <email address hidden>
Date: Fri Sep 4 15:10:49 2020 +0100

    Fix CA file for API client

    If a custom CA file is configured via OPENSTACK_SSL_CACERT, currently
    communication with Keystone will fail, since the session is not created
    using this CA file.

    This change fixes the issue by passing the path to the CA file to the
    keystoneauth session constructor.

    Change-Id: Iad1bdea97ed649cc3c8f042dc5dd147b989dfd0e
    Closes-Bug: #1873736

Reviewed: https://review.opendev.org/752426
Committed: https://git.openstack.org/cgit/openstack/masakari-monitors/commit/?id=1b47056bb6b7f9d7a13dfe5f00c642652e39dfe2
Submitter: Zuul
Branch: stable/stein

commit 1b47056bb6b7f9d7a13dfe5f00c642652e39dfe2
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 19:02:28 2020 +0000

    Use keystoneauth1 config option loading for masakari client

    If a custom CA file is configured via [api] cafile, currently
    communication with Keystone will fail, since the session is not created
    using this CA file. The [api] insecure option is also ignored.

    This change fixes the issue by using keystoneauth loading for the auth
    and session, to ensure all standard configuration options are supported.

    Change-Id: Idd58b72f7f5242e8135fec71b42adf5dd1852417
    Closes-Bug: #1873736
    (cherry picked from commit e70404588002222c8475667ff1a94f4c265a8def)

tags: added: in-stable-stein
tags: added: in-stable-ussuri

Reviewed: https://review.opendev.org/752424
Committed: https://git.openstack.org/cgit/openstack/masakari-monitors/commit/?id=e23f06f865e08ab4c75c9cd6bc723a934ec85663
Submitter: Zuul
Branch: stable/ussuri

commit e23f06f865e08ab4c75c9cd6bc723a934ec85663
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 19:02:28 2020 +0000

    Use keystoneauth1 config option loading for masakari client

    If a custom CA file is configured via [api] cafile, currently
    communication with Keystone will fail, since the session is not created
    using this CA file. The [api] insecure option is also ignored.

    This change fixes the issue by using keystoneauth loading for the auth
    and session, to ensure all standard configuration options are supported.

    Change-Id: Idd58b72f7f5242e8135fec71b42adf5dd1852417
    Closes-Bug: #1873736
    (cherry picked from commit e70404588002222c8475667ff1a94f4c265a8def)

tags: added: in-stable-train

Reviewed: https://review.opendev.org/752425
Committed: https://git.openstack.org/cgit/openstack/masakari-monitors/commit/?id=4d13e712cbb4139c6e04fe4a3b00a4072d195eb1
Submitter: Zuul
Branch: stable/train

commit 4d13e712cbb4139c6e04fe4a3b00a4072d195eb1
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 19:02:28 2020 +0000

    Use keystoneauth1 config option loading for masakari client

    If a custom CA file is configured via [api] cafile, currently
    communication with Keystone will fail, since the session is not created
    using this CA file. The [api] insecure option is also ignored.

    This change fixes the issue by using keystoneauth loading for the auth
    and session, to ensure all standard configuration options are supported.

    Change-Id: Idd58b72f7f5242e8135fec71b42adf5dd1852417
    Closes-Bug: #1873736
    (cherry picked from commit e70404588002222c8475667ff1a94f4c265a8def)

Reviewed: https://review.opendev.org/749203
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=1dd743aed317c2550fbd089cf13f518c9fec7aba
Submitter: Zuul
Branch: stable/ussuri

commit 1dd743aed317c2550fbd089cf13f518c9fec7aba
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 18:57:21 2020 +0000

    Fix CA file for nova client

    If a custom CA file is configured via nova_ca_certificates_file,
    currently communication with Keystone will fail, since the session is
    not created using this CA file. The same is true for nova_api_insecure.

    This change fixes the issue by using a keystoneauth session loader.

    Closes-Bug: #1873736

    Change-Id: I54a4f398a06c61e0f6f8f3efd3e4b6214bca756b
    (cherry picked from commit 3bd82c776c91bdd1caee474f3fc9cda18927da7f)

Reviewed: https://review.opendev.org/749204
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=fc7546a2aed061f661fc4e975343f32a6fdc27d8
Submitter: Zuul
Branch: stable/train

commit fc7546a2aed061f661fc4e975343f32a6fdc27d8
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 18:57:21 2020 +0000

    Fix CA file for nova client

    If a custom CA file is configured via nova_ca_certificates_file,
    currently communication with Keystone will fail, since the session is
    not created using this CA file. The same is true for nova_api_insecure.

    This change fixes the issue by using a keystoneauth session loader.

    Closes-Bug: #1873736

    Change-Id: I54a4f398a06c61e0f6f8f3efd3e4b6214bca756b
    (cherry picked from commit 3bd82c776c91bdd1caee474f3fc9cda18927da7f)

TODO(myself): this should be targeted at masakari-dashboard as well

Reviewed: https://review.opendev.org/754352
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=cd4c562eaf0bc5e5b8bc819aed5d35b11393e65e
Submitter: Zuul
Branch: stable/stein

commit cd4c562eaf0bc5e5b8bc819aed5d35b11393e65e
Author: Mark Goddard <email address hidden>
Date: Fri Jul 17 18:57:21 2020 +0000

    Fix CA file for nova client

    If a custom CA file is configured via nova_ca_certificates_file,
    currently communication with Keystone will fail, since the session is
    not created using this CA file. The same is true for nova_api_insecure.

    This change fixes the issue by using a keystoneauth session loader.

    Closes-Bug: #1873736

    Change-Id: I54a4f398a06c61e0f6f8f3efd3e4b6214bca756b
    (cherry picked from commit 3bd82c776c91bdd1caee474f3fc9cda18927da7f)
    (cherry picked from commit fc7546a2aed061f661fc4e975343f32a6fdc27d8)

Reviewed: https://review.opendev.org/753714
Committed: https://git.openstack.org/cgit/openstack/masakari-dashboard/commit/?id=9952f9e7a64811a7fcf1e604110b443132525567
Submitter: Zuul
Branch: stable/ussuri

commit 9952f9e7a64811a7fcf1e604110b443132525567
Author: Mark Goddard <email address hidden>
Date: Fri Sep 4 15:10:49 2020 +0100

    Fix CA file for API client

    If a custom CA file is configured via OPENSTACK_SSL_CACERT, currently
    communication with Keystone will fail, since the session is not created
    using this CA file.

    This change fixes the issue by passing the path to the CA file to the
    keystoneauth session constructor.

    Change-Id: Iad1bdea97ed649cc3c8f042dc5dd147b989dfd0e
    Closes-Bug: #1873736
    (cherry picked from commit 59ee1d59a1e08cd2245a8e876ea8931419ac5e32)

Reviewed: https://review.opendev.org/753715
Committed: https://git.openstack.org/cgit/openstack/masakari-dashboard/commit/?id=d4c4665d00b1229d1a408a0eb67148c298753c8f
Submitter: Zuul
Branch: stable/train

commit d4c4665d00b1229d1a408a0eb67148c298753c8f
Author: Mark Goddard <email address hidden>
Date: Fri Sep 4 15:10:49 2020 +0100

    Fix CA file for API client

    If a custom CA file is configured via OPENSTACK_SSL_CACERT, currently
    communication with Keystone will fail, since the session is not created
    using this CA file.

    This change fixes the issue by passing the path to the CA file to the
    keystoneauth session constructor.

    Change-Id: Iad1bdea97ed649cc3c8f042dc5dd147b989dfd0e
    Closes-Bug: #1873736
    (cherry picked from commit 59ee1d59a1e08cd2245a8e876ea8931419ac5e32)

Reviewed: https://review.opendev.org/753716
Committed: https://git.openstack.org/cgit/openstack/masakari-dashboard/commit/?id=2fdf12524bef53fad5c5ecd3e360956eb2d5aabd
Submitter: Zuul
Branch: stable/stein

commit 2fdf12524bef53fad5c5ecd3e360956eb2d5aabd
Author: Mark Goddard <email address hidden>
Date: Fri Sep 4 15:10:49 2020 +0100

    Fix CA file for API client

    If a custom CA file is configured via OPENSTACK_SSL_CACERT, currently
    communication with Keystone will fail, since the session is not created
    using this CA file.

    This change fixes the issue by passing the path to the CA file to the
    keystoneauth session constructor.

    Change-Id: Iad1bdea97ed649cc3c8f042dc5dd147b989dfd0e
    Closes-Bug: #1873736
    (cherry picked from commit 59ee1d59a1e08cd2245a8e876ea8931419ac5e32)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers