Signing error: Error opening signer certificate /etc/keystone/ssl/certs/signing_cert.pem

[Anastasia Palkina]

"build_id": "2014-09-25_00-01-46",
"ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9",
"build_number": "527",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "47275134ee3a97442937c4862037dfba04b8cc69",
"production": "docker",
"fuelmain_sha": "df29118860ab7c2d8f0f5f7ca805032c87716645",
"astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
"feature_groups": ["mirantis"],
"release": "6.0",
"release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-25_00-01-46", "ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9", "build_number": "527", "api": "1.0", "nailgun_sha": "47275134ee3a97442937c4862037dfba04b8cc69", "production": "docker", "fuelmain_sha": "df29118860ab7c2d8f0f5f7ca805032c87716645", "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "ae297a55b7ee2958fa2cd5bab3ce0175f0818ea6"}}},
"fuellib_sha": "ae297a55b7ee2958fa2cd5bab3ce0175f0818ea6"

1. Create new environment (CentOS, simple mode)
2. Choose nova-network, flat manager
3. Choose both Ceph
4. Choose rados
5. Add 1 controller, 1 compute, 3 ceph
6. Start deployment. It was successful
7. Start OSTF tests. It was successful
8. But there are many errors on controller (node-13) in keystone-all.log:

2014-09-26 09:41:45 ERROR

keystone.common.wsgi [-] Command 'openssl' returned non-zero exit status 3
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 212, in __call__
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi result = method(context, **params)
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/controller.py", line 152, in inner
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi return f(self, context, *args, **kwargs)
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 443, in revocation_list
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi CONF.signing.keyfile)
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystoneclient/common/cms.py", line 242, in cms_sign_text
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi raise subprocess.CalledProcessError(retcode, "openssl")
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3
2014-09-26 08:41:45.356 20467 TRACE keystone.common.wsgi

2014-09-26 09:41:45 ERROR

keystoneclient.common.cms [-] Signing error: Error opening signer certificate /etc/keystone/ssl/certs/signing_cert.pem
140201453164360:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/keystone/ssl/certs/signing_cert.pem','r')
140201453164360:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
unable to load certificate

[Stanislaw Bogatkin (sbogatkin)]

We need newer package python-keystoneclient, cause it has been fixed in 0.9.0: https://bugs.launchpad.net/keystone/+bug/1312858

[Dmitry Mescheryakov (dmitrymex)]

We've already merged keystone client 0.11, hence the issue must be fixed

[Anastasia Palkina (apalkina)]

Reproduced on ISO #112

"build_id": "2014-11-17_22-00-23",
"ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068",
"build_number": "112",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "da81b8f6812e39582be8162f87252f93e043fa34",
"production": "docker",
"fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2",
"astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896",
"feature_groups": ["mirantis"],
"release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-17_22-00-23", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "112", "api": "1.0", "nailgun_sha": "da81b8f6812e39582be8162f87252f93e043fa34", "production": "docker", "fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2", "astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "8a0ceff90777af75a3f9363a57185e608f3ee10d"}}},
"fuellib_sha": "8a0ceff90777af75a3f9363a57185e608f3ee10d"

1. Create new environment (CentOS, simple mode)
2. Choose nova-network, flat manager
3. Choose both Ceph
4. Choose rados
5. Add 1 controller, 1 compute, 3 ceph
6. Start deployment. It was successful
7. Start OSTF tests. It was successful
8. But there are many errors on controller (node-1) in /var/log/keystone/keystone-all.log:

2014-11-18 11:40:53.936 21487 ERROR keystoneclient.common.cms [-] Signing error: Error opening signer certificate /etc/keystone/ssl/certs/signing_cert.pem
2014-11-18 11:40:53.939 21487 ERROR keystoneclient.common.cms [-] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
2014-11-18 11:40:53.940 21487 ERROR keystone.common.wsgi [-] Command 'openssl' returned non-zero exit status 3

[Boris Bobrov (bbobrov)]

Firstly, I still cannot reliably reproduce the bug. It is reproduced on freshly-deployed instance, but after some time it magically disappeares.

After some investigation, the issue was localized to middleware, auth_token.py:AuthProtocol._validate_user_token. Sometimes pki-related methods are called despite uuid provider being set in config.

IMO, the issue is somehow related to cache. Will check that tomorrow.

[Boris Bobrov (bbobrov)]

A set of simillar bugreports already exist for this issue:
http://tracker.ceph.com/issues/9493
https://bugzilla.redhat.com/show_bug.cgi?id=1141615 (please read all the comments there)

A fix to Puppet scripts was proposed and accepted -- https://review.openstack.org/#/c/123547/

[Vladimir Kuklin (vkuklin)]

I see, that the only issue for now are artifacts in keystone logs. Lowering down the severity and pushing to 6.1 release

[Boris Bobrov (bbobrov)]

http://paste.openstack.org/show/137449/ this is part of log showing the caller and revocation failure effect, btw.

[Boris Bobrov (bbobrov)]

There is a patch https://review.openstack.org/#/c/131036/ that probably addresses this problem; it might be interesting to our Ceph people.

[Bogdan Dobrelya (bogdando)]

According to the Boris's comments #7 and #10 this issue looks like have to be addressed by MOS-Keystone team

[Boris Bobrov (bbobrov)]

No. The patch I mentioned in comment #10 probably fixes the bug. And it might or might not get to master.

In the comment #7 I give a link to a patch, that was already merged to puppet scripts, fixing the issue. Puppet scripts is not MOS-Keystone's area.

[Charlie Ott (charlieott)]

Hi. I am having this same issue with 3 controllers+mongo, and 4 compute+ceph nodes. deploying juno with Fuel 6.0 deployment.

<11>Mar 4 20:51:33 node-50 keystone-all Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
--
2015-03-04 20:51:33.227 6723 TRACE keystone.common.wsgi raise subprocess.CalledProcessError(retcode, 'openssl')
2015-03-04 20:51:33.227 6723 TRACE keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3
<11>Mar 4 20:52:33 node-50 keystone-all Signing error: Error opening signer certificate /etc/keystone/ssl/certs/signing_cert.pem
140441757452104:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/keystone/ssl/certs/signing_cert.pem','r')
140441757452104:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
<11>Mar 4 20:52:33 node-50 keystone-all Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
--
...repeats every 60 seconds.

Should I attempt to create a signing certificate and enable it with ceph as explained here: https://bugzilla.redhat.com/show_bug.cgi?id=1141615#c10

Or, should i just set the revocation timeout in /etc/ceph.conf to somthing ridiculously high, and ignore it until the next update?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172730

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172730
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0b0234f48abed0154e5ec9dcb11cd23470246035
Submitter: Jenkins
Branch: master

commit 0b0234f48abed0154e5ec9dcb11cd23470246035
Author: Bartłomiej Piotrowski <email address hidden>
Date: Sun Apr 12 19:59:51 2015 +0200

    keystone: pki-setup should be run regardless of token provider

    The puppet-keystone module currently will only execute 'keystone-manage pki_setup'
    to create the signing key if the PKI token provider is being used. The signing
    key/cert is still used for signing the token revocation list, even when the UUID
    token provider is being used. We should be running 'keystone-manage pki_setup'
    if enable_pki_setup=true, regardless of token provider.

    Closes-Bug: 1374349
    Upstream commit: I2817fbde74cbd50cae31f681503816e576cc7b60

    Change-Id: I293593194a7545aecd3ebd825d108b4b1c20ba29

[Anastasia Palkina (apalkina)]

Verified on ISO #323

"build_id": "2015-04-20_07-52-58", "ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847", "build_number": "323", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-20_07-52-58", "ostf_sha": "4bab9b975ace8d9a305d6e0f112b734de587f847", "build_number": "323", "api": "1.0", "nailgun_sha": "5a4556ea0cf943edffa33788994b7fb7abe150b3", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "991fa8feca4afcf05ce85a1e7ebd7c7503b3f212", "fuellib_sha": "36f30ae7f19092a61eebb0522ca20d27468b4cbf"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "5a4556ea0cf943edffa33788994b7fb7abe150b3", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "991fa8feca4afcf05ce85a1e7ebd7c7503b3f212", "fuellib_sha": "36f30ae7f19092a61eebb0522ca20d27468b4cbf"