Pike - keystone with SSL PKI unconfigured unable to load certificate

[Jeff Hillman]

From keystone.log

(keystoneclient.common.cms): 2018-03-08 23:20:50,811 ERROR Signing error: Unable to load certificate - ensure you have configured PKI with "
keystone-manage pki_setup"
(keystone.common.wsgi): 2018-03-08 23:20:50,812 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 228, in __call__
    result = method(req, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 94, in inner
    return f(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 350, in revocation_list
    CONF.signing.keyfile)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
    signing_key_file_name, message_digest=message_digest)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
    raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3

From the charm in keystone_hooks.py I see that it is supposed to skip this for Pike.

    if CompareOpenStackReleases(os_release('keystone-common')) >= 'pike':
        # pike dropped support for PKI token; skip function
        return

However the log seems to contradict this.

We are seeing a lack of usability with keystone with Pike + SSL. Without SSL this same environment was functional. The above is the only error in keystone. Currently the CLI and dashboard are not available.

[Ryan Beisner (1chb1n)]

Please include juju crashdump and/or the juju bundle, status and relevant juju unit logs. Thank you.

[Jeff Hillman (jhillman)]

juju status

https://pastebin.canonical.com/p/sVxQqMKhZY/

bundle

https://pastebin.canonical.com/p/8WpKfZN65R/

unit-keystone-0.log

[attached]

[Chris Gregan (cgregan)]

Escalated to Field Critical as is is actively blocking a in progress customer deploy without workaround

[Ryan Beisner (1chb1n)]

Let's get this onto the latest stable charms and make sure it is a current bug. Thank you.

[Ryan Beisner (1chb1n)]

FYI - It appears to be the 17.11 charms.

[Ryan Beisner (1chb1n)]

TIP: This comment is not directly related to the root cause of this bug, but you can now declare charms without series in the charm store url. We recommend doing so as all of the OpenStack-Charms projects support multi-series.

ex.

Simplify and use:
    charm: cs:keystone

Instead of the old format:
    charm: cs:xenial/keystone

[Ryan Beisner (1chb1n)]

I know you've destroyed and are about to redeploy with latest wares, and the following relates to the initial deploy referenced in this bug. But pasting observations thus-far:

The traceback in the uploaded file is completely different than that shown in the bug description. While it may be related, I'd like to see the unit log and keystone deamon logs for the same timeframe matching the traceback from the bug description (or a repro of that).

Also, I'm not able to determine what the .166 address is from the model status or bundles pastes, but this is a problem:
ssh: connect to host 10.243.8.166 port 22: Connection timed out

This could be a network or MTU issue. If the issue it recurs with port 22 connect timeouts, please confirm switchport MTUs are high enough for NIC MTUs and container interface MTUs.

https://pastebin.canonical.com/p/jGGZXbH2m6/

and

https://pastebin.canonical.com/p/zP9xZZcGps/

...

2018-03-09 15:34:03 INFO juju-log cluster:22: Error syncing remote files
2018-03-09 15:34:03 DEBUG cluster-relation-changed Traceback (most recent call last):
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 880, in <module>
2018-03-09 15:34:03 DEBUG cluster-relation-changed main()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 873, in main
2018-03-09 15:34:03 DEBUG cluster-relation-changed hooks.execute(sys.argv)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-09 15:34:03 DEBUG cluster-relation-changed self._hooks[hook_name]()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-09 15:34:03 DEBUG cluster-relation-changed restart_functions)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-09 15:34:03 DEBUG cluster-relation-changed r = lambda_f()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-09 15:34:03 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1650, in _inner_update_certs_if_available
2018-03-09 15:34:03 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 618, in cluster_changed
2018-03-09 15:34:03 DEBUG cluster-relation-changed update_all_identity_relation_units_force_sync()
2018-03-09 15:34:03 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", li...

[Ryan Beisner (1chb1n)]

FWIW, the MTU check is just based on seeing similar recent issues and has no other evidence as basis. ;-)

[Jeff Hillman (jhillman)]

Updated to 18.02 charms with identical bundle produced same error.

Setting enable-pki=true produced same error with now an error 500 for keystone requests.

created a smaller bundle that had just mysql and keystone and nothing else. enable-pke=False, use-https=True, https-service-endpoints=True puts it in a ready state.

However, now I'm getting certificate verify failed when running basic openstack endpoint list. I suspect there is an issue with the certs I was provided.

Will re-request certs and test full deploy.

[Vern Hart (vern)]

We received the new certs and they were exactly the same as the previous certs -- which suggests their certs were maybe not the problem.

The client's certificates were signed by a sub certificate authority which was signed by their root certificate. Previously we had the host cert in ssl_cert and the sub-ca cert in ssl_ca. The new certs included the cert for their root server so I did two things:

# I used their root cert for ssl_ca.
# I concatenated the sub-ca and root certs into the host cert files and used the new combined cert as ssl_cert. (cat old-cert.crt subca.crt root.crt > new-cert.crt)

After redeploying openstack bundle, things are working as expected.

To clarify, in addition to combining certs, we also have https-service-endpoints=True and use-https=True. And enable-pki is the default (False).

[James Page (james-page)]

I think you need to not use:

https-service-endpoints=True and use-https=True

This activates the self signed features in the keystone charm which I think
will take precedent over any provided using ssl* options.
On Sun, 11 Mar 2018 at 08:30, Vern Hart <email address hidden> wrote:

> We received the new certs and they were exactly the same as the previous
> certs -- which suggests their certs were maybe not the problem.
>
> The client's certificates were signed by a sub certificate authority
> which was signed by their root certificate. Previously we had the host
> cert in ssl_cert and the sub-ca cert in ssl_ca. The new certs included
> the cert for their root server so I did two things:
>
> # I used their root cert for ssl_ca.
> # I concatenated the sub-ca and root certs into the host cert files and
> used the new combined cert as ssl_cert. (cat old-cert.crt subca.crt
> root.crt > new-cert.crt)
>
> After redeploying openstack bundle, things are working as expected.
>
> To clarify, in addition to combining certs, we also have https-service-
> endpoints=True and use-https=True. And enable-pki is the default
> (False).
>
> --
> You received this bug notification because you are a member of Canonical
> Field Critical, which is subscribed to the bug report.
> Matching subscriptions: openstack-charms
> https://bugs.launchpad.net/bugs/1754682
>
> Title:
> Pike - keystone with SSL PKI unconfigured unable to load certificate
>
> Status in OpenStack keystone charm:
> Incomplete
>
> Bug description:
> From keystone.log
>
> (keystoneclient.common.cms): 2018-03-08 23:20:50,811 ERROR Signing
> error: Unable to load certificate - ensure you have configured PKI with "
> keystone-manage pki_setup"
> (keystone.common.wsgi): 2018-03-08 23:20:50,812 ERROR Command 'openssl'
> returned non-zero exit status 3
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line
> 228, in __call__
> result = method(req, **params)
> File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py",
> line 94, in inner
> return f(self, request, *args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py",
> line 350, in revocation_list
> CONF.signing.keyfile)
> File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py",
> line 336, in cms_sign_text
> signing_key_file_name, message_digest=message_digest)
> File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py",
> line 384, in cms_sign_data
> raise subprocess.CalledProcessError(retcode, 'openssl')
> CalledProcessError: Command 'openssl' returned non-zero exit status 3
>
>
> From the charm in keystone_hooks.py I see that it is supposed to skip
> this for Pike.
>
> if CompareOpenStackReleases(os_release('keystone-common')) >= 'pike':
> # pike dropped support for PKI token; skip function
> return
>
> However the log seems to contradict this.
>
> We are seeing a lack of usability with keystone with Pike + SSL.
> Without SSL this same environment was functional. The above is the
> only error in keystone. Currently the CLI ...

[Corey Bryant (corey.bryant)]

(re-added comment due to formatting issue)

Vern, Good to hear that things are working.

James, we should update the keystone charm's README if that is the case. It currently has:

"use-https - if enabled this option tells Keystone to configure the identity endpoint as https. Under this model the keystone charm will either use the CA as provided by the user (see ssl_* options below) or will generate its own and sync across peers. The cert will be distributed to all service endpoints which will be configured to use https."

"https-service-endpoints - if enabled this option tells Keystone to configure ALL endpoints as https. Under this model the keystone charm will either use the CA as provided by the user (see ssl_* options below) or will generate its own and sync across peers. The cert will be distributed to all service endpoints which will be configured to use https as well as configuring themselves to be used as https."

[James Page (james-page)]

OK so to confirm:

    # generate or get a new cert/key for service if set to manage certs.
    https_service_endpoints = config('https-service-endpoints')
    if https_service_endpoints and bool_from_string(https_service_endpoints):

is the code snippet from keystone - if this config option is enabled, keystone will act as a CA and generate certs and keys for all endpoints presented over relations - you *don't* want this to happen in a production cloud where certs, keys and ca's are presented via the ssl_* configuration options.

[James Page (james-page)]

(the README is inaccurate IMHO)

[James Page (james-page)]

That said, looking at the codepaths it would appear that using the ssl_* options should override any keystone charm generated certs and keys.

However I'd still not recommend using the charm this way.

[James Page (james-page)]

So to confirm:

 enable-pki=False
 https-service-endpoints=False
 use-https=False

*just* use the ssl_* options on each charm.

[James Page (james-page)]

Marking Low as this is really a doc issue and unsubscribing field-high.

[Vern Hart (vern)]

James: What's confusing to me is that in the original bundle, where we encountered this problem, we had not specified https-service-endpoints or use-https which means they were the default: False. We had ssl_ca, ssl_cert, and ssl_key defined but were getting the keystone charm trying to run keystone-manage pki_setup.

After some testing with Corey we narrowed down setting https-service-endpoints and use-https to True and that seemed to be enough to make the error go away. Services are using the defined certs as per the ssl_* options.

[Corey Bryant (corey.bryant)]

Hi Vern, I'm wondering if the ssl_* options weren't configured correctly with the original bundle, and possibly adding use-https/https-service-endpoints caused it to use a charm-generated CA. The error was a signing error coming from openssl. Exit code 3 defined here: https://www.openssl.org/docs/man1.1.0/apps/cms.html#EXIT-CODES.

Are you able to re-deploy with your updated ssl_* options and https-service-endpoints/use-https disabled?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/552106

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/552106
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=3384ddcb87a2c7621719c8844cf5a24a25936d48
Submitter: Zuul
Branch: master

commit 3384ddcb87a2c7621719c8844cf5a24a25936d48
Author: Corey Bryant <email address hidden>
Date: Mon Mar 12 14:15:26 2018 -0400

    Update SSL/https documentation

    The README documentation implies that use-https and
    https-service-endpoints are required when enabling SSL/https
    with your own CA, SSL cert, and key. Update the README and
    config.yaml to explain that config options use-https and
    https-service-endpoints should not be set when using ssl_*
    config options.

    Change-Id: I2e0140f909ef2c57182895f37cf191b6bc80157b
    Closes-Bug: #1754682

[Vern Hart (vern)]

I know this is committed but I wanted to confirm that things are still working after redeploying with ssl_{ca,cert,key} defined and use-https and https-service-endpoints false.

[Corey Bryant (corey.bryant)]

Thanks for letting us know Vern.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/555028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555028
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=3a4faf4df4794b6031b49079b2c56309a3578a7f
Submitter: Zuul
Branch: stable/18.02

commit 3a4faf4df4794b6031b49079b2c56309a3578a7f
Author: Corey Bryant <email address hidden>
Date: Mon Mar 12 14:15:26 2018 -0400

    Update SSL/https documentation

    The README documentation implies that use-https and
    https-service-endpoints are required when enabling SSL/https
    with your own CA, SSL cert, and key. Update the README and
    config.yaml to explain that config options use-https and
    https-service-endpoints should not be set when using ssl_*
    config options.

    Change-Id: I2e0140f909ef2c57182895f37cf191b6bc80157b
    Closes-Bug: #1754682
    (cherry picked from commit 3384ddcb87a2c7621719c8844cf5a24a25936d48)