tripleo

Incorrect service auth URL in Octavia

Bug #1824777 reported by Carlos Goncalves on 2019-04-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Carlos Goncalves	tripleo train-1

Bug Description

Creating a TERMINATED_HTTPS loadbalancer listener is failing with 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request.

The issue with creating a TERMINATED_HTTPS listener is that the auth_url in the [service_auth] section of octavia.conf is being configured by tripleo to use KeystonePublic as the endpoint and not KeystoneV3Public. Hand-editing octavia.conf and adding "/v3" to the auth_url, restarting the service, and running through the 'steps to reproduce' results in a successful loadbalancer listener creation.

2019-03-27 19:34:03.042 1 DEBUG octavia.certificates.manager.barbican [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Setting project ACL for certificate secret... set_acls /usr/lib/python2.7/site-packages/octavia/certificates/manager/barbican.py:148
2019-03-27 19:34:03.747 1 DEBUG barbicanclient.client [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Creating Client object Client /usr/lib/python2.7/site-packages/barbicanclient/client.py:156
2019-03-27 19:34:03.809 1 DEBUG barbicanclient.v1.acls [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Getting ACL for secret href: http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/acl get /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:458
2019-03-27 19:34:04.106 1 DEBUG wsme.api [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222

In an attempt to get a more meaningful error message, file /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py was modified and was able to get it to generate this traceback:

2019-03-28 04:44:31.080 1 ERROR root [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] : NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root Traceback (most recent call last):
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py", line 333, in load_acls_data
2019-03-28 04:44:31.080 1 ERROR root response = self._api.get(self.acl_ref)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 70, in get
2019-03-28 04:44:31.080 1 ERROR root return super(_HTTPClient, self).get(*args, **kwargs).json()
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'GET', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 62, in request
2019-03-28 04:44:31.080 1 ERROR root resp = super(_HTTPClient, self).request(*args, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request
2019-03-28 04:44:31.080 1 ERROR root return self.session.request(url, method, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request
2019-03-28 04:44:31.080 1 ERROR root auth_headers = self.get_auth_headers(auth)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2019-03-28 04:44:31.080 1 ERROR root return auth.get_headers(self, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2019-03-28 04:44:31.080 1 ERROR root token = self.get_token(session)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2019-03-28 04:44:31.080 1 ERROR root return self.get_access(session).auth_token
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-03-28 04:44:31.080 1 ERROR root self.auth_ref = self.get_auth_ref(session)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root return self._plugin.get_auth_ref(session, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root authenticated=False, log=False)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post
2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'POST', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request
2019-03-28 04:44:31.080 1 ERROR root raise exceptions.from_response(resp, method, url)
2019-03-28 04:44:31.080 1 ERROR root NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root
2019-03-28 04:44:31.152 1 DEBUG barbicanclient.v1.acls [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] After _api.get load_acls_data /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:336
2019-03-28 04:44:31.162 1 DEBUG wsme.api [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222

Original report: https://bugzilla.redhat.com/show_bug.cgi?id=1693808

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-15: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/652603

Changed in tripleo:
assignee:	nobody → Carlos Goncalves (cgoncalves)
status:	New → In Progress

Alex Schultz (alex-schultz) on 2019-06-03

Changed in tripleo:
importance:	Undecided → Medium
milestone:	none → train-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/652603
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9f6caf7724fb6529a838595a5a9d4607d2eebace
Submitter: Zuul
Branch: master

commit 9f6caf7724fb6529a838595a5a9d4607d2eebace
Author: Carlos Goncalves <email address hidden>
Date: Mon Apr 15 11:00:29 2019 +0200

Fix service auth URL in Octavia

This patch changes the service auth URL to use Keystone v3 internal
endpoint.

Closes-Bug: #1824777
Change-Id: I4d70ad67df24950a943006d4793d7c6bc6b50f18

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/662938

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/662940

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/662941

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/662938
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=517175cce086e0a46d63a185390e44dd088cf659
Submitter: Zuul
Branch: stable/stein

commit 517175cce086e0a46d63a185390e44dd088cf659
Author: Carlos Goncalves <email address hidden>
Date: Mon Apr 15 11:00:29 2019 +0200

Fix service auth URL in Octavia

This patch changes the service auth URL to use Keystone v3 internal
endpoint.

    Closes-Bug: #1824777
    Change-Id: I4d70ad67df24950a943006d4793d7c6bc6b50f18
    (cherry picked from commit 9f6caf7724fb6529a838595a5a9d4607d2eebace)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/662940
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0663f3feadc71fe26988ee3ab61db7a4fde940de
Submitter: Zuul
Branch: stable/rocky

commit 0663f3feadc71fe26988ee3ab61db7a4fde940de
Author: Carlos Goncalves <email address hidden>
Date: Mon Apr 15 11:00:29 2019 +0200

Fix service auth URL in Octavia

This patch changes the service auth URL to use Keystone v3 internal
endpoint.

    Closes-Bug: #1824777
    Change-Id: I4d70ad67df24950a943006d4793d7c6bc6b50f18
    (cherry picked from commit 9f6caf7724fb6529a838595a5a9d4607d2eebace)
    (cherry picked from commit 517175cce086e0a46d63a185390e44dd088cf659)

tags:	added: in-stable-rocky
tags:	added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/662941
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9934640fc266fa5d64e2aa9a0b6d2edfe2f4a64d
Submitter: Zuul
Branch: stable/queens

commit 9934640fc266fa5d64e2aa9a0b6d2edfe2f4a64d
Author: Carlos Goncalves <email address hidden>
Date: Mon Apr 15 11:00:29 2019 +0200

Fix service auth URL in Octavia

This patch changes the service auth URL to use Keystone v3 internal
endpoint.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-07: Fix included in openstack/tripleo-heat-templates 11.0.0

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-21: Fix included in openstack/tripleo-heat-templates 9.4.0

#10

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-21: Fix included in openstack/tripleo-heat-templates 8.4.0

#11

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-05: Fix included in openstack/tripleo-heat-templates 10.6.0

#12

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.