[SRU] keystoneclient fails on SSL certificates that work for other services

Bug #1064835 reported by Sina Sadeghi
48
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
python-keystoneclient
Fix Released
Critical
Sam Morrison
python-keystoneclient (Ubuntu)
Fix Released
Critical
Unassigned
Quantal
Fix Released
Critical
Unassigned

Bug Description

[IMPACT]

Crticial. Enabling SSL for services in the Keystone catalog does not function with certain types of certificates, eg wild card certificates.

[TESTCASE]

Put services in the keystone catalog behind a service that termintaes SSL with wild card certificates, and the client will receive SSL errors.

[Regression Potential]

Minimal. The one-line patch simply fixes the argument handling of the Keystone client to ensure SSL connection work correctly in all cases.

Hi guys,

I've successfully deployed and tested keystone and wanted to stick some SSL in front of it. I used the nginx puppet module provided here https://github.com/NeCTAR-RC/puppet-nginx and seemed to successfully deploy the module.

I've updated all my endpoints to be https:// and also updated my "auth.rc" file for the same.

When I run "keystone user-list" I get the following error

Unable to communicate with identity service: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib. (HTTP 400)

When I run "keystone --insecure user-list" I get the correct output, so I know the SSL is working ok and I'm affected by a verification issue.

Same when using keystone-init:

# ./keystone-init.py config.yaml
No handlers could be found for logger "keystoneclient.client"
Traceback (most recent call last):
  File "./keystone-init.py", line 97, in <module>
    configure_keystone(config)
  File "./keystone-init.py", line 28, in configure_keystone
    default_tenant = create_default_tenant(keystone, config)
  File "./keystone-init.py", line 43, in create_default_tenant
    enabled=True)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/tenants.py", line 83, in create
    return self._create('/tenants', params, "tenant")
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 82, in _create
    resp, body = self.api.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/client.py", line 174, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/client.py", line 155, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/client.py", line 135, in request
    raise exceptions.from_response(resp, body)
keystoneclient.exceptions.BadRequest: Unable to communicate with identity service: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed. (HTTP 400)

...and if I modify keystone-init.py to add insecure=True to the client call, it works fine.

However this is where I'm stumped, as the certificate I'm using is a wildcard *.example.com style certificate, and it works on all the other services (email, apache, etc) which deploy it for.

I also note that if I point my browser to https://keystone-int.example.com:5000 that my Mozilla claims the certificate is correctly verified by our provider (GoDaddy).

I'd really like a better understanding of why the keystoneclient thinks my cert isn't verified when Firefox is fine with it. As far as I can see, the GoDaddy certs are present in both /etc/ssl/certs as well as in the cacerts.txt file of python-httplib2 and I added our private key into /etc/ssl/private. So it would be unlikely that we are missing an intermediate certificate. Using --insecure is all well and good for the CLI, but what about the keystone stuff in swift, nova and and glance? Will I have to specify insecure everywhere and ask our clients to use --insecure?

Would really appreciate some help around this issue as it's a major roadblock to moving forward.

Revision history for this message
Sina Sadeghi (sina-sa) wrote :

Screenshot showing side-by-side firefox and keystoneclient, one can verify the other can't.

Running on Ubuntu 12.04 LTS with the Ubuntu Cloud-Archive Folsom repo in use.

Revision history for this message
Sam Morrison (sorrison) wrote :

This is a bug in the ubuntu cloud archive

Changed in python-keystoneclient:
status: New → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in keystone (Ubuntu):
status: New → Confirmed
Sam Morrison (sorrison)
tags: added: cloud-archive
Revision history for this message
Sam Morrison (sorrison) wrote :

On further investigation it is a problem in keystone, whoops

Changed in keystone (Ubuntu):
status: Confirmed → Invalid
Changed in python-keystoneclient:
status: Invalid → Confirmed
Revision history for this message
Sam Morrison (sorrison) wrote :

OK I have tracked this down. It looks to me that keystone-client is unusable with an an https keystone endpoint.

If --os-cacert is not set it will default to an empty string. This then gets passed down to underlying http classes and it will use the empty string as the ca path.

What needs to happen is that is this option is not set it needs to be set as None.

That way httplib will use the default CA path but only if it is None, None != "" for this checking.

tags: added: folsom-backport-potential
removed: cloud-archive
Revision history for this message
Alan Pevec (apevec) wrote :

So does the following fix this issue?

--- a/keystoneclient/shell.py
+++ b/keystoneclient/shell.py
@@ -131,7 +131,7 @@ class OpenStackIdentityShell(object):

         parser.add_argument('--os-cacert',
                             metavar='<ca-certificate>',
- default=env('OS_CA_CERT'),
+ default=env('OS_CA_CERT', default=None),
                             help='Defaults to env[OS_CACERT]')
         parser.add_argument('--os_cacert',
                             help=argparse.SUPPRESS)

or more generic:

--- a/keystoneclient/shell.py
+++ b/keystoneclient/shell.py
@@ -42,7 +42,7 @@ def env(*vars, **kwargs):
         value = os.environ.get(v, None)
         if value:
             return value
- return kwargs.get('default', '')
+ return kwargs.get('default', None)

 class OpenStackIdentityShell(object):

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-keystoneclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/14572

Changed in python-keystoneclient:
assignee: nobody → Sam Morrison (sorrison)
status: Confirmed → In Progress
Revision history for this message
Alan Pevec (apevec) wrote : Re: keystoneclient fails on SSL certificates that work for other services

Removing tag for backport, keystoneclient doesn't have stable/folsom branch, all clients are trunk-only now (for better or worse).

tags: removed: folsom-backport-potential
Sam Morrison (sorrison)
affects: keystone (Ubuntu) → python-keystoneclient (Ubuntu)
Changed in python-keystoneclient (Ubuntu):
status: Invalid → Confirmed
tags: added: canonistack
Revision history for this message
Sam Morrison (sorrison) wrote :

Is there anything holding this up?
keystoneclient is practically useless in an https environment without it. Would be good to push this along.
I would mark it as critical if I could

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

WRT Ubuntu update, this needs to be fixed upstream before it can be applied as a match to the Ubuntu package in Quantal and Cloud Archive.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-keystoneclient (master)

Reviewed: https://review.openstack.org/14572
Committed: http://github.com/openstack/python-keystoneclient/commit/d81160e32e9828c6ba59da28a1f5ed79c9d05fdf
Submitter: Jenkins
Branch: master

commit d81160e32e9828c6ba59da28a1f5ed79c9d05fdf
Author: Sam Morrison <email address hidden>
Date: Sat Oct 20 05:05:17 2012 +1100

    Fixes https connections to keystone when no CA certificates
    are specified.

    Fixes bug 1064835

    Change-Id: I77987f01cc1395a4857ebdcb0bb6adf697866dd7

Changed in python-keystoneclient:
status: In Progress → Fix Committed
Sam Morrison (sorrison)
tags: added: cloud-archive
summary: - keystoneclient fails on SSL certificates that work for other services
+ [SRU] keystoneclient fails on SSL certificates that work for other
+ services
description: updated
Changed in python-keystoneclient (Ubuntu Quantal):
status: New → Confirmed
Changed in python-keystoneclient (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Sina Sadeghi (sina-sa) wrote :

Thanks devs?

Is there a rough ETA as to when this fix will end up in the cloud archive?

Revision history for this message
Scott Kitterman (kitterman) wrote :

Patch is missing in the quantal upload. Here's the full diff:

diff -Nru python-keystoneclient-0.1.3/debian/changelog python-keystoneclient-0.1.3/debian/changelog
--- python-keystoneclient-0.1.3/debian/changelog 2012-09-07 22:32:42.000000000 +0000
+++ python-keystoneclient-0.1.3/debian/changelog 2012-10-31 10:32:26.000000000 +0000
@@ -1,3 +1,11 @@
+python-keystoneclient (1:0.1.3-0ubuntu1.1) quantal-proposed; urgency=low
+
+ * debian/patches/fix-https-with-no-CA.patch: Cherry-picked upstream fix that
+ ensures HTTPS client connections work when no CA are specified.
+ (LP: #1064835)
+
+ -- Adam Gandelman <email address hidden> Wed, 31 Oct 2012 09:08:17 +0100
+
 python-keystoneclient (1:0.1.3-0ubuntu1) quantal; urgency=low

Rejecting. Please upload again.

Michael Still (mikal)
Changed in python-keystoneclient:
importance: Undecided → Critical
Changed in python-keystoneclient (Ubuntu):
importance: Undecided → Critical
Changed in python-keystoneclient (Ubuntu Quantal):
importance: Undecided → Critical
Revision history for this message
Dave Walker (davewalker) wrote : Please test proposed package

Hello Sina, or anyone else affected,

Accepted into quantal-proposed. The package will build now and be available in a few hours in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-keystoneclient (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Sina Sadeghi (sina-sa) wrote :
Download full text (3.6 KiB)

I updated the package from quantal-proposed successfully. I checked the contents of shell.py to ensure the patch had been applied and removed all *.pyc files from /usr/lib/python2.7/dist-packages/keystoneclient/ to ensure no files from the previous version remained.

Unfortunately, it seems the bug still persists. I cannot interact with keystone using SSL, unless I use the --insecure flag, or specify the default cacerts.txt file by hand. Please see below.

# dpkg -l python-keystoneclient
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-====================================-====================================-========================================================================================
ii python-keystoneclient 1:0.1.3-0ubuntu1.1 Client libary for Openstack Keystone API

# grep -B 4 CACERT /usr/share/pyshared/keystoneclient/shell.py

        parser.add_argument('--os-cacert',
                            metavar='<ca-certificate>',
                            default=env('OS_CA_CERT', default=None),
                            help='Defaults to env[OS_CACERT]')

# keystone service-list
No handlers could be found for logger "keystoneclient.client"
Unable to communicate with identity service: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed. (HTTP 400)

# keystone --insecure service-list
+----------------------------------+----------+--------------+------------------------------+
| id | name | type | description |
+----------------------------------+----------+--------------+------------------------------+
| 26b4ab0f46904930bc907ddf8204f09b | volume | volume | Nova Volume Service |
| 39d9a621022543c3aecc52734080725e | glance | image | Glance Image Service |
| 5b5d4d9edd2d45fab7a64a284349988c | swift | object-store | Swift Object Storage Service |
| 769f19f0d056482988558d6c611f2df8 | nova | compute | Nova Compute Service |
| a59de4515aae4e36b7a94ed2ad008fbe | ec2 | ec2 | EC2 Compatibility Layer |
| c77e502d8a8e42b18cfed9a6c4d35b9e | keystone | identity | Keystone Identity Service |
+----------------------------------+----------+--------------+------------------------------+

# keystone --os-cacert /usr/share/pyshared/httplib2/cacerts.txt service-list
+----------------------------------+----------+--------------+------------------------------+
| id | name | type | description |
+----------------------------------+----------+--------------+------------------------------+
| 26b4ab0f46904930bc907ddf8204f09b | volume | volume | Nova Volume Service |
| 39d9a621022543c3aecc52734080725e | glance | image | Glance Image Service |
| 5b5d4d9edd2d45fab7a64a284349988c | swift | object-store | Swift Object Stor...

Read more...

tags: added: verification-failed
removed: verification-needed
Revision history for this message
Dave Walker (davewalker) wrote :

This is being discussed if it really did fail verification. There seems to be some uncertainty if the same bug is being discussed. Thanks

Revision history for this message
Sam Morrison (sorrison) wrote :

I can confirm this fixes the issue.

On ubuntu it will use /etc/ssl/certs/ca-certificates.crt which is specified in httplib2/__init__.py

Revision history for this message
Sina Sadeghi (sina-sa) wrote :

Tag update to reflect that the bug has been fixed.

tags: added: verification-done
removed: verification-failed
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-keystoneclient - 1:0.1.3-0ubuntu1.1

---------------
python-keystoneclient (1:0.1.3-0ubuntu1.1) quantal-proposed; urgency=low

  * debian/patches/fix-https-with-no-CA.patch: Cherry-picked upstream fix that
    ensure HTTPS client connections work when no CA are specified.
    (LP: #1064835)
 -- Michael Still <email address hidden> Mon, 12 Nov 2012 10:27:28 +0000

Changed in python-keystoneclient (Ubuntu Quantal):
status: Fix Committed → Fix Released
Revision history for this message
James Page (james-page) wrote :

Fixed in 1:0.1.3-0ubuntu1.1~cloud0

Changed in cloud-archive:
status: New → Fix Released
Dolph Mathews (dolph)
Changed in python-keystoneclient:
milestone: none → 0.2.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.