keystoneclient appears to ignore --os-cacert option

[Eoghan Glynn]

I expected to be able to enable the keystone service for HTTPS access and then specify the CA cert on the client side. However keystoneclient seems not to take the --os-cacert into account as expected.

Steps to reproduce:

# create keystone certs

$ keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone
$ chown keystone:keystone /etc/keystone/ssl/certs/*

# in /etc/keystone/keystone.conf, set the following:

[ssl]
enable = True
certfile = /etc/keystone/ssl/certs/keystone.pem
keyfile = /etc/keystone/ssl/private/keystonekey.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
ca_key = /etc/keystone/ssl/certs/cakey.pem

# re-create identity service endpoint with https URLs:

$ KEYSTONE_SERVICE=$(keystone service-list | awk '/keystone/ {print $2}')
$ KS_IP=keystone_host_ip
$ keystone endpoint-create --region RegionOne --service-id=$KEYSTONE_SERVIVE --publicurl=https://$KS_IP:5000/v2.0 --internalurl=https://$KS_IP:5000/v2.0 --adminurl=https://$KS_IP:35357/v2.0

# delete old keystone endpoint

$ keystone --insecure endpoint-delete OLD_KEYSTONE_ENDPOINT

# restart keystone

$ sudo service openstack-keystone restart # or equivalent on Ubuntu

# run keystone client insecurely (i.e. just confidentiality on the wire):

$ keystone --insecure endpoint-list | awk '/35357/ {print $10}'
   https://172.16.12.49:35357/v2.0

# attempt to specify the CA cert so that we the service-side cert can be verified, fails:

$ keystone --debug --os-cacert /etc/keystone/ssl/certs/ca.pem endpoint-list
DEBUG:keystoneclient.session:REQ: curl -i -X POST https://172.16.12.49:35357/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "123ca675e5ce45f6"}}}'
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): 172.16.12.49
Authorization Failed: SSL exception connecting to https://172.16.12.49:35357/v2.0/tokens

# use the equivalent -k option to curl, and it works:

$ curl -k /etc/keystone/ssl/certs/ca.pem -i -X POST https://172.16.12.49:35357/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "123ca675e5ce45f6"}}}'
curl: (3) <url> malformed
HTTP/1.1 200 OK
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 8988
Date: Thu, 13 Mar 2014 23:20:35 GMT
{"access": {"token": {"issued_at": "2014-03-13T23:20:35.668186", "expires": "2014-03-14T23:20:35Z", "id": ...

Is my understanding of --os-cacert option correct?

If not, how is the equivalent behavior enabled?

[Jamie Lennox (jamielennox)]

So i've tried this setup with both installing certificates manually and with ssl_setup and the behaviour of --insecure/--os-cacert works for me with both.

The curl command appears to be going to the correct endpoints which is always my first guess. Don't worry about the fact the curl statement is missing the certificate files - that is a debugging statement only.

In curl -k is --insecure, i'm not sure if that's a mistake in the bug report or your curl command is also doing an insecure connection. That would help explain it if it's not.

Other things to check when setting up ssl, you should set:

[ssl]
cert_subject=/C=US/ST=Unset/L=Unset/O=Unset/CN=172.16.12.49

if you are going to use ssl_setup so that the hostname is correct.

[Adam Huffman (adam-huffman)]

Adding my own experience here, which sounds very similar, using Icehouse. Here's my keystone invocation:

keystone --os-cacert /etc/grid-security/certificates/UKeScienceCA-2B.pem --debug endpoint-list

and here's the debugging output:

DEBUG:keystoneclient.session:REQ: curl -i -X POST https://<hostname>:35357/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "<password>"}}}'
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): <hostname>
Authorization Failed: SSL exception connecting to https://<hostname>:35357/v2.0/tokens

which looks to me as though the curl command generated doesn't specify the CA certificate.

If I run that curl command, it fails, of course. If I run it like this:

curl -v --cacert /etc/grid-security/certificates/UKeScienceCA-2B.pem -i -X POST https://<hostname>:35357/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "<password>"}}}'

it succeeds, as expected. This behaviour looks to me to replicate the original report i.e. the --os-cacert parameter being ignored.

[Dolph Mathews (dolph)]

Is the issue that the cert is being ignored by keystoneclient, or that it's being accidentally excluded from the example curl command? Not that keystoneclient itself is *not* issuing commands to curl.

[Adam Huffman (adam-huffman)]

The primary issue for me is that the cert is being ignored. The secondary issue is, I suppose, whether the example curl command reflects what keystoneclient is actually doing. I.e. I know it's not *actually* running curl, but does the example curl command reflect the equivalent of what it *is* actually doing, without any reference to the cert.

[Adam Huffman (adam-huffman)]

I also see the same behaviour with the nova and glance clients.

[Adam Huffman (adam-huffman)]

Actually, the problem for me was that I needed to specify a different CA certificate (we're using e-Science certificates), after which it all started working.