'openssl' returned non-zero exit status 4 w/ SSL enabled

[Lorin Hochstein]

From this thread on the openstack-operators mailing list: http://lists.openstack.org/pipermail/openstack-operators/2013-May/002903.html

Version: grizzly

If SSL is enabled in keystone, cinder command-line client commands don't work.

Changing the following settings in keystone.conf and restarting resolves the issue:

[ssl]
enable = False

[signing]
token_format = UUID

Error in cinder-api.log is:

2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token] Token
validation failure.
Traceback (most recent call last):
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 688, in _validate_user_token
    verified = self.verify_signed_token(user_token)
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1043, in verify_signed_token
    if self.is_signed_token_revoked(signed_text):
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1007, in is_signed_token_revoked
    revocation_list = self.token_revocation_list
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1079, in token_revocation_list
    self.token_revocation_list = self.fetch_revocation_list()
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1109, in fetch_revocation_list
    return self.cms_verify(data['signed'])
  File
"/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py",
line 1038, in cms_verify
    raise err
CalledProcessError: Command 'openssl' returned non-zero exit status 4
*2013-04-30 20:00:42 DEBUG [keystoneclient.middleware.auth_token]
Marking token *MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzY
...
i4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
*as unauthorized in memcache*
*2013-04-30 20:00:42 WARNING [keystoneclient.middleware.auth_token]
Authorization failed for
token*MIIMbwYJKoZIhvcNAQcCoIIMYDCCDFwCAQExCTAHBgUrDgMCGjCCC0gGCSqGSIb3DQEHAaCCCzkEggs1eyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNC0zMFQyMDowMDo0Mi40MDYzNTMiLCAiZXhwaXJlcyI6ICIyMDEzLTA1LTAxVDIwOjAwOjQyWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6ODc3NC92Mi82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTciLCAiaWQiOiAiMjYxNzgzOTEyNzVhNDJjZmEzYjc4NmFiMTUxYzhmOGEiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjg3NzQvdjIvNmFhM2JmMWFiNjgwNDAyMTg4NzNhNzgyZjkwY2ZmYTcifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuY
...
dmljZXMvQ2xvdWQifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiZWMyIiwgIm5hbWUiOiAiZWMyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MSIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjExOjgwODAvdjEvQVVUSF82YWEzYmYxYWI2ODA0MDIxODg3M2E3ODJmOTBjZmZhNyIsICJpZCI6ICI2NTkxMTExNGMzNjM0MWExOTAwNmMzMjhjNmQwYTJhZSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6ODA4MC92MS9BVVRIXzZhYTNiZjFhYjY4MDQwMjE4ODczYTc4MmY5MGNmZmE3In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm9iamVjdC1zdG9yZSIsICJuYW1lIjogInN3aWZ0In0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTE6MzUzNTcvdjIuMCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTkuMTM2LjEwOjUwMDAvdjIuMCIsICJpZCI6ICIwZjkzODlkMDQ4NWU0ZjJmOWY3ODc0YzQxMTgxYmQyOCIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xOS4xMzYuMTA6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJhZG1pbiIsICJyb2xlc19saW5rcyI6IFtdLCAiaWQiOiAiM2Y4MjY3M2I1ZmUwNDExYWI1ZmQ4MjE2YmRiNjkzYzYiLCAicm9sZXMiOiBbeyJuYW1lIjogIktleXN0b25lU2VydmljZUFkbWluIn0sIHsibmFtZSI6ICJLZXlzdG9uZUFkbWluIn0sIHsibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNjY2NmZhOTkwNzhhNGYwN2EwNzBlN2U4NThjMzJmMDIiLCAiMzZiYmE5ZWYwMTc4NDQ4YzhhNjU0Yjc1ZmViM2EwZjQiLCAiYTI1NTgxZGQzNDcwNDYwYjkxZWNhYTI5ZWNhNzIwNWMiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCbzuXTFZ8vZ2h4VnLUvdrzn5HCJdeEI5KkpLLHLkVvjrYwPm6NC+sRvDZ0Mg2MCMHtt1eK4o0GRBtmq8sTtUGqHuT5Ns41whp+r+diTGNfkW6mOaJBwpQhxbjXiTGcCHWJni3RkDTDinY-O7Zto3ct0etVmxvE62lqSFSQUKoyAg==
*2013-04-30 20:00:42 INFO [keystoneclient.middleware.auth_token]
Invalid user token - rejecting request*
*
*

[Carlos (c-e-g)]

I found the same problem in the package keystone. I'm using ubuntu 12.04.2 with Grizzly repository and configuring Glance. Applied this workaround and it worked.

[Vincent Hou (houshengbo)]

I was trying with the latest code of OpenStack.
When ssl is enabled in Keystone, cinder command is unable to run, neither is nova command.

======================================Keystone Log==========================================
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/opt/stack/keystone/keystone/common/wsgi.py", line 164, in _run
    log=WritableLogger(log))
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 655, in server
    client_socket = sock.accept()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 277, in accept
    suppress_ragged_eofs=self.suppress_ragged_eofs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/ssl.py", line 46, in __init__
    super(GreenSSLSocket, self).__init__(sock.fd, *args, **kw)
  File "/usr/lib/python2.7/ssl.py", line 141, in __init__
    ciphers)
SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
Removing descriptor: 7
===========================================================================================

[Dolph Mathews (dolph)]

I imagine that disabling SSL isn't a viable "workaround" for a lot of people.. what's the actual bug here? Was keystone-manage pki_setup run successfully?

[Dolph Mathews (dolph)]

I suspect keystoneclient.middleware.auth_token in front of glance & cinder aren't configured to validate PKI tokens? I really doubt this is an issue with cinder/glance.

[Launchpad Janitor (janitor)]

[Expired for Keystone because there has been no activity for 60 days.]

[Eduard Barrera (eduard-barrera)]

I had the same problem using RedHat RDO

[sngirame (sngirame)]

I also faced the same problem using RDO Grizzly on centOS & even the workaround did not help.
nova image-list or glance image-list comand works without any issue. However, cinder list command fails with ERROR: Unauthorized

[wingwj (wingwj)]

I met the same problem. The issue is caused by the jumping time.
My sys-time jumps out the valid openssl-cert time span, and caused this problem.

After I revert the system time, my environment returns normal.
Of course you can recreate the cert-file to recover.

Hope it's useful to you.

[Vikas Deolaliker (vikasd)]

I ran into this problem with IH-RC1. Adjusting system time worked.

[David Hill (david-hill-ubisoft)]

I have this issue at this time and the time is OK. Glance, Nova and keystone work fine but cinder always return 401. I've recreated the SSL certificate and PKI keys ... same behavior.

[David Hill (david-hill-ubisoft)]

I'm trying to migrate from havana to icehouse and I have this cinder issue.

[David Hill (david-hill-ubisoft)]

One more note:

[signing]
 token_format = UUID

This is the only modification I need to do in order to restore the cinder service.

[Dolph Mathews (dolph)]

Unassigning due to inactivity.

[Adam Young (ayoung)]

Please try the steps here.

http://adam.younglogic.com/2013/07/troubleshooting-pki-middleware/

[Adam Huffman (adam-huffman)]

I went through those steps and things still weren't working. Changing back to UUID tokens allowed me to get further.

[guomin.lizte (limin6886)]

I am troubling by this problerm, keystone works fine but nova always return 401; nova-api.log hints keystoneclient.middleware.auth_token [-] Verify error: Command 'openssl' returned non-zero exit status 4; the only way to avoid it is the same as #12

[Steve Heyman (sheyman)]

I had this same problem in Barbican due to expired certs (couldn't fetch the revocation list). It wasn't clear without going in with debugger and looking at returned JSON. Sounds like some serviceability fixes could be in order here?

[Steve Heyman (sheyman)]

My problem is that the signing cert from the keystone endpoint I'm using has expired. I can see this using curl <host:port>/v2.0/certificates/signing

[Lin Yang (lin-a-yang)]

I met the same problem with Juno on Ubuntu14.04. After adjusted the system time, it works.

[Steve Martinelli (stevemar)]

based on the comments above, adjusting the system time seems like the right answer, and there isn't much we can do from a code perspective to fix it, this is just part of the issues and headaches that PKI tokens have caused over the years with their configuration.

on the same note, we will not be pursuing any PKI token work as of Mitaka because we deprecating that format. marking this as invalid.