GET /os-quota-sets/{tenant_id} API is failing with SSL exception

[prashkre]

In the flow of GET /os-quota-sets/{tenant_id} API, when project_id/tenant_id is being verified by communicating with keystone through secure(https)connection at https://github.com/openstack/nova/blob/master/nova/api/openstack/identity.py#L32, it is failing in certificate validation error as below.

2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity Traceback (most recent call last):
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/nova/api/openstack/identity.py", line 42, in verify_project_id
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity raise_exc=False)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 758, in get
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity return self.request(url, 'GET', **kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity return wrapped(*args, **kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity resp = send(**kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 678, in _send_request
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity raise exceptions.SSLError(msg)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://xxx.xxx.xxx.xxx:5000/v3/projects/0fe761dc32934fc88c390d244acb6971: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity

[Matthew Edmonds (edmondsw)]

The problem is that https://github.com/openstack/nova/blob/f6fbfc7ff07b790ef052a759552c69429b3d79c7/nova/api/openstack/identity.py#L32 simply relies on the default for the Session constructor's verify argument (True). We need to provide a configuration option to allow an operator to specify verify=False and verify=<path>.

[Matthew Edmonds (edmondsw)]

this presumably also impacts POST /flavors/{flavor_id}/action with addTenantAccess or removeTenantAccess actions, since they also call verify_project_id. E.g.: https://github.com/openstack/nova/blob/4d552d6bf61955fa121feedbb4469f4ed0b07cff/nova/api/openstack/compute/flavor_access.py#L99

[jichenjc (jichenjc)]

we rely on this for nova to communicate with keystone, and this bug also occurs when nova api talking to keystone, so the cred should be able to be reused?

[keystone_authtoken]
auth_type = v3password
auth_url = xxxx
auth_uri = xxx
username = nova
project_name = service
user_domain_id = default
project_domain_id = default
cafile = /data/PKIcerts/certs/cacert.pem
password = xxxx

so if we pass cafile in this definition into code from
sess = session.Session(auth=context.get_auth_plugin())
to
sess = session.Session(auth=context.get_auth_plugin(), cert=CONF.keystone_authtoken.cafile)

should be fine?

[Matthew Edmonds (edmondsw)]

it's not what that conf section is for, but yeah, I like that idea. We should avoid creating more conf options than we have to, and the cert needed for this verify_project_id would always match the one needed in that section, so creating a new conf option would just be duplicating things unnecessarily.

[jichenjc (jichenjc)]

ok, maybe need some testing, I remember we can't create https in devstack env, can we ? or could you please help to verify if this change could solve the problem you mentioned?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/485121

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485121
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=905d31dd9715505599b0a2ad123eebef37f606f5
Submitter: Jenkins
Branch: master

commit 905d31dd9715505599b0a2ad123eebef37f606f5
Author: jichenjc <email address hidden>
Date: Sun Jul 9 10:35:23 2017 +0800

    Enable custom certificates for keystone communication

    Nova creates a session back to keystone to verify project ids for
    quota and flavor access APIs. The session that was being created was
    not based on conf options, so it only worked in simple default
    scenarios. This updates the session by using the newly added keystone
    section to utilize keystoneauth1 to manage the session creation, which
    allows for specifying custom site certificates to secure the link
    between Nova and Keystone.

    Change-Id: Ice4b226fdabdfb66e60b61de05ac8f3b37610661
    Closes-Bug: 1704798

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/505546

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505546
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c0ff5b33ca42a417a5fa08e5133a27397af08c3d
Submitter: Zuul
Branch: stable/pike

commit c0ff5b33ca42a417a5fa08e5133a27397af08c3d
Author: jichenjc <email address hidden>
Date: Sun Jul 9 10:35:23 2017 +0800

    Enable custom certificates for keystone communication

    Nova creates a session back to keystone to verify project ids for
    quota and flavor access APIs. The session that was being created was
    not based on conf options, so it only worked in simple default
    scenarios. This updates the session by using the newly added keystone
    section to utilize keystoneauth1 to manage the session creation, which
    allows for specifying custom site certificates to secure the link
    between Nova and Keystone.

    Change-Id: Ice4b226fdabdfb66e60b61de05ac8f3b37610661
    Closes-Bug: 1704798
    (cherry picked from commit 905d31dd9715505599b0a2ad123eebef37f606f5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.2 release.