functional tests broken in ssl environment

[Brian Rosmaita]

This happens when running the functional tests with a devstack with 'enable_service tls-proxy' in the local.conf

Here's an example traceback:
=== start ===
Traceback (most recent call last):
  File "glanceclient/tests/functional/test_readonly_glance.py", line 28, in test_list_v1
    out = self.glance('--os-image-api-version 1 image-list')
  File "glanceclient/tests/functional/base.py", line 66, in glance
    **kwargs)
  File "/home/rosmabr/working/git/openstack/python-glanceclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 205, in glance
    'glance', action, flags, params, fail_ok, merge_stderr)
  File "/home/rosmabr/working/git/openstack/python-glanceclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 396, in cmd_with_auth
    self.cli_dir, prefix=self.prefix)
  File "/home/rosmabr/working/git/openstack/python-glanceclient/.tox/functional/local/lib/python2.7/site-packages/tempest/lib/cli/base.py", line 71, in execute
    result_err)
tempest.lib.exceptions.CommandFailed: Command '['/home/rosmabr/working/git/openstack/python-glanceclient/.tox/functional/bin/glance', '--os-username', 'admin', '--os-tenant-name', 'admin', '--os-password', 'ChangeMe', '--os-auth-url', 'https://172.16.132.143/identity', '--os-user-domain-id', 'default', '--os-project-domain-id', 'default', '--os-endpoint-type', 'publicURL', '--os-image-api-version', '1', 'image-list']' returned non-zero exit status 1.
stdout:

stderr:
Unable to determine the Keystone version to authenticate with using the given auth_url. Identity service may not support API version discovery. Please provide a versioned auth_url instead. error=SSL exception connecting to https://172.16.132.143/identity: HTTPSConnectionPool(host='172.16.132.143', port=443): Max retries exceeded with url: /identity (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
=== end ===

The "Unable to determine the Keystone version ..." message is ours, it's generated when the ewxception is caught. The real reason for the failure is given in the 'error=' part.

[Brian Rosmaita (brian-rosmaita)]

I was looking through the devstack tls-proxy code and noticed the function fix_system_ca_bundle_path: https://github.com/openstack-dev/devstack/blob/master/lib/tls#L343-L369

Roughly, the python requests module uses its own CA bundle rather that the distro's bundle, where the devstack certs have been added, and during devstack install, devstack fixes this. When the functional tests are run against devstack, tox creates its own virtualenv that contains the requests module that will be used by the tests, and I think this requests module is using the wrong bundle, which is why the tests fail.

I did by hand what fix_system_ca_bundle_path does for the python in .tox/functional/bin/python and then ran the functional tests, and this time they passed.

I guess what we need to do is to modify the tox ini so that it calls fix_system_ca_bundle_path before the tests run.

[Brian Rosmaita (brian-rosmaita)]

To be clear about what I did to get the tests to pass:

I'm using ubuntu.

$ tox -e functional --notest
$ source .tox/functional/bin/activate
$ capath=$(python -c $'try:\n from requests import certs\n print (certs.where())\nexcept ImportError: pass')
$ rm -f $capath
$ ln -s /etc/ssl/certs/ca-certificates.crt $capath
$ tox -e functional

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/528384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/528384
Committed: https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=a5985508817e5de73092a339f15ce7f9f701a20f
Submitter: Zuul
Branch: master

commit a5985508817e5de73092a339f15ce7f9f701a20f
Author: Brian Rosmaita <email address hidden>
Date: Fri Dec 15 14:35:49 2017 -0500

    Restore functional testing under ssl

    Closes-bug: #1738033
    Change-Id: Ia3e2e210eea09ac07311f25ffa99ad0f4ced418d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-glanceclient 2.9.0 release.