Unable to generate CA and launch cluster

Bug #1717798 reported by Avtar singh
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Magnum
Fix Released
Undecided
Luca Lorenzetto

Bug Description

I am Deploying Kubernetes cluster on Openstack PIKE and using fedora-atom image. But unable to do so as I launched the cluster it gets timeout error after 60 mints. After checking I found out that the certificates are not generating which I narrowed it down to a CURL call unable to generate CA certificate.

==================
#curl -k -X GET \
 # -H "X-Auth-Token: $USER_TOKEN" \
 # -H "OpenStack-API-Version: container-infra latest" \
 # $MAGNUM_URL/certificates/$CLUSTER_UUID | python -c 'import sys, json; print json.load(sys.stdin)["pem"]' > ${CA_CERT}
==================

It is giving the following error while i run it manually
====================
{"errors": [{"status": 500, "code": "server", "links": [], "title": "Remote error: BadRequest Invalid input for field 'identity/password/user/password': None is not of type 'string'\n\nFailed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n {'type': 'string'}\n\nOn instance['identity']['password']['user']['password']:\n None (HTTP 400) (Request-ID: req-e9e666aa-70d9-41b8-86ac-f80fe0fcf4c7)\n[u'", "detail": "Remote error: BadRequest Invalid input for field 'identity/password/user/password': None is not of type 'string'\n\nFailed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n {'type': 'string'}\n\nOn instance['identity']['password']['user']['password']:\n None (HTTP 400) (Request-ID: req-e9e666aa-70d9-41b8-86ac-f80fe0fcf4c7)\n[u'", "request_id": ""}]}[

====================

In magnum logs error looks like this
=====================
Sep 17 12:50:51 controller magnum-api: Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
Sep 17 12:50:51 controller magnum-api: {'type': 'string'}
Sep 17 12:50:51 controller magnum-api: On instance['identity']['password']['user']['password']:
Sep 17 12:50:51 controller magnum-api: None (HTTP 400) (Request-ID: req-ab4870e8-c18c-47c3-9abb-b2c8338d7e5b)
Sep 17 12:50:51 controller magnum-api: [u'Traceback (most recent call last):\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/conductor/handlers/indirection_api.py", line 33, in _object_dispatch\n return getattr(target, method)(context, *args, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/objects/cluster.py", line 141, in get_by_uuid\n db_cluster = cls.dbapi.get_cluster_by_uuid(context, uuid)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/db/sqlalchemy/api.py", line 212, in get_cluster_by_uuid\n query = self._add_tenant_filters(context, query)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/db/sqlalchemy/api.py", line 141, in _add_tenant_filters\n user_name = kst.client.users.get(context.user_id).name\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneclient/v3/users.py", line 152, in get\n user_id=base.getid(user))\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func\n return f(*args, **new_kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneclient/base.py", line 349, in get\n self.key)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneclient/base.py", line 150, in _get\n resp, body = self.client.get(url, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get\n return self.request(url, \'GET\', **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 463, in request\n resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request\n return self.session.request(url, method, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-pa
Sep 17 12:50:51 controller magnum-api: ckages/keystoneauth1/session.py", line 573, in request\n auth_headers = self.get_auth_headers(auth)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers\n return auth.get_headers(self, **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers\n token = self.get_token(session)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 89, in get_token\n return self.get_access(session).auth_token\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 135, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref\n authenticated=False, log=False, **rkwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post\n return self.request(url, \'POST\', **kwargs)\n', u' File "/var/lib/magnum/env/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request\n raise exceptions.from_response(resp, method, url)\n', u"BadRequest: Invalid input for field 'identity/password/user/password': None is not of type 'string'\n\nFailed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:\n {'type': 'string'}\n\nOn instance['identity']['password']['user']['password']:\n None (HTTP 400) (Request-ID: req-ab4870e8-c18c-47c3-9abb-b2c8338d7e5b)\n"].". Detail:
Sep 17 12:50:51 controller magnum-api: Traceback (most recent call last):
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
Sep 17 12:50:51 controller magnum-api: result = f(self, *args, **kwargs)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/api/controllers/v1/certificate.py", line 145, in get_one
Sep 17 12:50:51 controller magnum-api: cluster = api_utils.get_resource('Cluster', cluster_ident)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/api/utils.py", line 97, in get_resource
Sep 17 12:50:51 controller magnum-api: return resource.get_by_uuid(pecan.request.context, resource_ident)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 182, in wrapper
Sep 17 12:50:51 controller magnum-api: args, kwargs)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/objects/base.py", line 76, in object_class_action
Sep 17 12:50:51 controller magnum-api: objver, args, kwargs)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/magnum/conductor/api.py", line 73, in object_class_action
Sep 17 12:50:51 controller magnum-api: objver=objver, args=args, kwargs=kwargs)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 465, in call
Sep 17 12:50:51 controller magnum-api: return self.prepare().call(ctxt, method, **kwargs)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
Sep 17 12:50:51 controller magnum-api: retry=self.retry)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
Sep 17 12:50:51 controller magnum-api: timeout=timeout, retry=retry)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
Sep 17 12:50:51 controller magnum-api: retry=retry)
Sep 17 12:50:51 controller magnum-api: File "/var/lib/magnum/env/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
Sep 17 12:50:51 controller magnum-api: raise result
Sep 17 12:50:51 controller magnum-api: RemoteError: Remote error: BadRequest Invalid input for field 'identity/password/user/password': None is not of type 'string'
Sep 17 12:50:51 controller magnum-api: Failed validating 'type' in schema['properties']['identity']['properties']['password']['properties']['user']['properties']['password']:
Sep 17 12:50:51 controller magnum-api: {'type': 'string'}
Sep 17 12:50:51 controller magnum-api: On instance['identity']['password']['user']['password']:
Sep 17 12:50:51 controller magnum-api: None (HTTP 400) (Request-ID: req-ab4870e8-c18c-47c3-9abb-b2c8338d7e5b)

====================
I hope the provided logs help.

Revision history for this message
yatin (yatinkarel) wrote :

@Avtar, some queries and suggestion below:-
1) how you installed magnum?
2) Can you check your magnum.conf [keystone_authtoken] section contains admin_user, admin_password and admin_tenant_name, If they are not present add them and try again after restarting magnum services.

Revision history for this message
Avtar singh (avtarsingh12015) wrote :

@yatin
1.) I have installed magnum using official documentation https://docs.openstack.org/magnum/pike/install/install-guide-from-source.html

2.) Keystone credentials are already present in the magnum.conf.

Revision history for this message
yatin (yatinkarel) wrote :

@avtar, https://docs.openstack.org/magnum/pike/install/install-guide-from-source.html doesn't contain
admin_user, admin_password and admin_tenant_name in [keystone_authtoken] section. Can you try setting these in [keystone_authtoken], you can also connect on IRC #openstack-containers

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to magnum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/504867

Changed in magnum:
assignee: nobody → yatin (yatinkarel)
status: New → In Progress
Changed in magnum:
assignee: yatin (yatinkarel) → Luca Lorenzetto (lorenzetto-luca)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to magnum (master)

Reviewed: https://review.openstack.org/504867
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=67a48749c78854657fd55e8125a1dcad11b2f67f
Submitter: Jenkins
Branch: master

commit 67a48749c78854657fd55e8125a1dcad11b2f67f
Author: yatin <email address hidden>
Date: Mon Sep 18 12:56:03 2017 +0530

    Fix magnum TLS cert generation

    TLS enabled k8s cluster are not created if admin_user, admin_password
    and admin_tenant_name are not set in [keystone_authtoken] section,
    This patch updates the install guide for the same.

    Change-Id: Id58bbbda34d52f60c174630400ae469c48a7d115
    Closes-Bug: #1717798
    Related-Bug: #1670355

Changed in magnum:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to magnum (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/509979

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to magnum (stable/pike)

Reviewed: https://review.openstack.org/509979
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=7944417ce35a4bc54292c1abd1d045c449f07b54
Submitter: Zuul
Branch: stable/pike

commit 7944417ce35a4bc54292c1abd1d045c449f07b54
Author: yatin <email address hidden>
Date: Mon Sep 18 12:56:03 2017 +0530

    Fix magnum TLS cert generation

    TLS enabled k8s cluster are not created if admin_user, admin_password
    and admin_tenant_name are not set in [keystone_authtoken] section,
    This patch updates the install guide for the same.

    Change-Id: Id58bbbda34d52f60c174630400ae469c48a7d115
    Closes-Bug: #1717798
    Related-Bug: #1670355
    (cherry picked from commit 67a48749c78854657fd55e8125a1dcad11b2f67f)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/magnum 6.0.0

This issue was fixed in the openstack/magnum 6.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/magnum 5.0.2

This issue was fixed in the openstack/magnum 5.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.