Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"

[Christoph Fiehe]

I have a fresh installation of OpenStack Newton based on Ubuntu 16.04. I am using Ceph Object Gateway as object storage implementation which regularly makes the following call "GET http://controller:5000/v3/auth/tokens/OS-PKI/revoked".

This call causes the following exception in the log of Keystone:
2016-10-20 14:30:33.764 13934 INFO keystone.common.wsgi [req-fccd6064-2c29-4929-8a68-8b439db14957 924990606827451ca0599a5dcc8fb2ec 76e3b8253287442bac2772138583cde9 - default default] GET http://os-identity:5000/v3/auth/tokens/OS-PKI/revoked
2016-10-20 14:30:33.889 13934 ERROR keystoneclient.common.cms [req-fccd6064-2c29-4929-8a68-8b439db14957 924990606827451ca0599a5dcc8fb2ec 76e3b8253287442bac2772138583cde9 - default default] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi [req-fccd6064-2c29-4929-8a68-8b439db14957 924990606827451ca0599a5dcc8fb2ec 76e3b8253287442bac2772138583cde9 - default default] Command 'openssl' returned non-zero exit status 3
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 225, in __call__
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi result = method(req, **params)
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 164, in inner
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs)
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 590, in revocation_list
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi CONF.signing.keyfile)
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 325, in cms_sign_text
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi signing_key_file_name, message_digest=message_digest)
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 373, in cms_sign_data
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi raise subprocess.CalledProcessError(retcode, 'openssl')
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3
2016-10-20 14:30:33.890 13934 ERROR keystone.common.wsgi

This is my keystone.conf:

[DEFAULT]
debug = false
# NOTE: log_dir alone does not work for Keystone
log_file = /var/log/keystone/keystone.log
transport_url = rabbit://keystone:XYZ@os-rabbit01:5672,keystone:XYZ@os-rabbit02:5672/openstack

[assignment]
driver = sql

[cache]
backend = oslo_cache.memcache_pool
enabled = true
memcache_servers = os-memcache:11211

[credential]
provider = fernet
key_repository = /etc/keystone/credential-keys

[database]
connection = mysql+pymysql://keystone:XYZ@os-controller/keystone
max_retries = -1

[memcache]
servers = os-memcache:11211

[oslo_messaging_notifications]
driver = messagingv2

[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1

[oslo_middleware]
enable_proxy_headers_parsing = true

[token]
driver = sql
provider = uuid

[extra_headers]
Distribution = Ubuntu

I know that with the Newton release a lot of things have been changed regarding signing and PKI. How can calls to Keystone's revocation list be handled in the Newton release without a PKI setup?

[Steve Martinelli (stevemar)]

We did not remove any of the PKI in newton, it was deprecated in Mitaka and didn't change much (at all?) for Newton. We will be removing it on Ocata.

Silly question, is openssl installed?

[Christoph Fiehe (fiehe)]

Yes, OpenSSL is installed. I got it working by setting up the PKI using keystone-manage pki_setup. Thank you very much for the hint that it get be removed in Ocata. The question is now: How does Keystone's endpoint reflect this change? The Ceph RADOS gateway queries periodally the revoked token list via "v3/auth/tokens/OS-PKI/revoked" that is signed with Keystone's private key. Is this signature completely removed or replaced by another mechanism? Please have a look here: http://docs.ceph.com/docs/jewel/radosgw/keystone/

[Lance Bragstad (lbragstad)]

Sounds like the issue was resolved. I'm going to mark this as Invalid since there doesn't seem to be a bug. Please feel free to continue using this bug for discussion if needed.

[Hamza (h16mara)]

Hi,

Since PKI is now removed from Ocata, can you please tell us what is the equivalent option to get the required certificate for the Ceph RADOS gateway as mentioned above ?

[sapd (saphi070)]

Hi @Hamza!
Did you have any ideas on Openstack Ocata version?

[Lance Bragstad (lbragstad)]

The `keystone-manage pki_setup` command is still available in the Ocata source [0]. But, we've never encouraged the use of that command to configure certificates for production use [1].

[0] https://github.com/openstack/keystone/blob/stable/ocata/keystone/cmd/cli.py#L539
[1] https://github.com/openstack/keystone/blob/stable/ocata/keystone/cmd/cli.py#L556-L557

[Christian Zunker (christian-zunker)]

Even in Pike, the error message with
> ensure you have configured PKI with "keystone-manage pki_setup"
still shows up in the logs.

The RedHat bug report suggest to disable revocation fetching:
https://bugzilla.redhat.com/show_bug.cgi?id=1438965

This worked in our case.

[Christian Zunker (christian-zunker)]

In the long run, this could help:
http://tracker.ceph.com/issues/19499