mojo dependencies cause SSL: CERTIFICATE_VERIFY_FAILED failure websocket-client

[Ryan Beisner]

If one creates a fresh python virtualenv and installs mojo, the version of websocket-client (0.44.0) causes mojo to fail as below.

If one restricts that version to 0.40.0, it succeeds.

Mojo should re-evaluate it's declared python dependencies.

Example:

virtualenv foo
. foo/bin/activate
pip install bzr bzr+lp:codetree#egg=codetree bzr+lp:mojo#egg=mojo websocket-client

...

(foo) jenkins@juju-0d10ff-osci-16:~/tools/z/charm-test-infra$ pip freeze
argcomplete==1.9.2
asn1crypto==0.22.0
Babel==2.5.0
bzr==2.7.0
certifi==2017.7.27.1
cffi==1.10.0
chardet==3.0.4
cryptography==2.0.3
debtcollector==1.17.0
enum34==1.1.6
funcsigs==1.0.2
futures==3.1.1
idna==2.6
ipaddress==1.0.18
iso8601==0.1.12
Jinja2==2.9.6
juju-deployer==0.10.0
jujuclient==0.54.0
keystoneauth1==3.1.0
MarkupSafe==1.0
mojo==0.4.5
monotonic==1.3
netaddr==0.7.19
netifaces==0.10.6
oslo.i18n==3.17.0
oslo.utils==3.28.0
pbr==3.1.1
pkg-resources==0.0.0
positional==1.2.1
prettytable==0.7.2
pycparser==2.18
pylxd==2.0.5
pyparsing==2.2.0
python-cinderclient==3.1.0
python-codetree==0.1.5
python-dateutil==2.6.1
pytz==2017.2
PyYAML==3.12
requests==2.18.4
requests-unixsocket==0.1.5
simplejson==3.11.1
six==1.10.0
stevedore==1.25.0
urllib3==1.22
websocket-client==0.40.0
wrapt==1.10.11
ws4py==0.4.2

...

https://pastebin.canonical.com/197065/

2017-08-29 22:08:20 [ERROR] There was an unrecognised problem with running a deploy phase
2017-08-29 22:08:20 [ERROR] Error found during deployment phase
2017-08-29 22:08:19 [DEBUG] deployer.cli: Using runtime GoEnvironment on auto-osci-sv07:admin/auto-osci-sv07
2017-08-29 22:08:19 [INFO] deployer.cli: Starting deployment of xenial-mitaka
2017-08-29 22:08:19 [DEBUG] deployer.import: Getting charms...
2017-08-29 22:08:19 [DEBUG] deployer.deploy: Resolving configuration
2017-08-29 22:08:20 [DEBUG] deployer.env: Connecting to auto-osci-sv07:admin/auto-osci-sv07...
Traceback (most recent call last):
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/bin/juju-deployer", line 11, in <module>
    sys.exit(main())
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/deployer/cli.py", line 140, in main
    run()
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/deployer/cli.py", line 250, in run
    importer.Importer(env, deployment, options).run()
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/deployer/action/importer.py", line 289, in run
    self.env.connect()
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/deployer/env/go.py", line 78, in connect
    self.client = self.client_class.connect(self.name)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/environment.py", line 87, in connect
    return connector().run(cls, env_name)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/connector.py", line 56, in run
    cert_path, data.get('environ-uuid'))
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/connector.py", line 64, in connect_env
    env = cls(endpoint, name=name, ca_cert=cert_path, env_uuid=env_uuid)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/juju2/environment.py", line 15, in __init__
    super(Environment, self).__init__(*args, **kw)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/environment.py", line 41, in __init__
    endpoint, self._ca_cert)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/jujuclient/connector.py", line 99, in connect_socket
    endpoint, origin=endpoint, sslopt=sslopt)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/websocket/_core.py", line 487, in create_connection
    websock.connect(url, **options)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/websocket/_core.py", line 211, in connect
    options.pop('socket', None))
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/websocket/_http.py", line 77, in connect
    sock = _ssl_socket(sock, options.sslopt, hostname)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/websocket/_http.py", line 182, in _ssl_socket
    sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
  File "/var/lib/jenkins/tools/z/charm-test-infra/.tox/clients/local/lib/python2.7/site-packages/websocket/_http.py", line 160, in _wrap_sni_socket
    server_hostname=hostname,
  File "/usr/lib/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 830, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

jenkins@juju-0d10ff-osci-16:~/tools/z/ubuntu-openstack-ci/run/job-parts$ juju version
2.2.2-xenial-amd64
jenkins@juju-0d10ff-osci-16:~/tools/z/ubuntu-openstack-ci/run/job-parts$ juju controllers
Use --refresh flag with this command to see the latest information.

Controller Model User Access Cloud/Region Models Machines HA Version
auto-osci-sv07* auto-osci-sv07 admin superuser serverstack/serverstack 2 1 none 2.2.2

jenkins@juju-0d10ff-osci-16:~/tools/z/ubuntu-openstack-ci/run/job-parts$ juju models
Controller: auto-osci-sv07

Model Cloud/Region Status Machines Cores Access Last connection
auto-osci-sv07* serverstack/serverstack available 0 - admin 5 seconds ago
controller serverstack/serverstack available 1 4 admin just now
default serverstack/serverstack available 0 - admin 2017-08-25

[Ryan Beisner (1chb1n)]

FYI, this dependency issue is coming from jujuclient:

Collecting websocket-client>=0.18.0 (from jujuclient>=0.0.7->mojo)
  Using cached websocket_client-0.44.0-py2.py3-none-any.whl

[Stuart Bishop (stub)]

Tagged python-jujuclient, as it appears to be incompatible with the latest websocket_client.

[Felipe Reyes (freyes)]

bundletester pinned websocket-client to <=0.40.0 https://github.com/juju-solutions/bundletester/commit/78b7b5b2707d1f6eaa507b38b33a1eccf715c3d9

[Casey Marshall (cmars)]

Is there a workaround for this? I seem to be unable to get mojo working on bionic, getting this same error.

[Daniel Manrique (roadmr)]

This is a horrible workaround, please don't use it.

If your environment allows (mine does, mojo deploying to a same-machine Juju controller on a VM inside a closed network, so it's super isolated), you can disable SSL cert checking and move ahead:

sudo vim /usr/lib/python3/dist-packages/jujuclient/connector
.py

look for def connect_socket(... and set sslopt['cert_reqs'] to ssl.CERT_NONE. The method should look like this after the surgery:

    def connect_socket(cls, endpoint, cert_path=None):
        """Return a websocket connection to an endpoint."""

        sslopt = cls.get_ssl_config(cert_path)
        sslopt['cert_reqs'] = ssl.CERT_NONE # I added this line
        return websocket.create_connection(
            endpoint, origin=endpoint, sslopt=sslopt)

Mojo should work after this.

To repeat, this is horrid because it entirely disables cert verification :( so please don't do it.