charm deployment fails, when using self-signed certificate, which has IP address only (SAN)

[Gábor Mészáros]

[Impact]
Bug 1771988 introduced a fix to support IP based SAN's in certificates; however the required new dependency (python-ipaddress) was not added to the Recommends of the package which was an oversight of the original SRU. This really only impacts on trusty deployments as on xenial python-ipaddress is installed indirectly via another dependency.

[Test Case]
apt install python-urllib3
python-ipaddress is not installed, certs with IP based SAN's won't verify correctly.

[Regression Potential]
Minimal - extra package installed on upgrades or install of urllib3

[Original Bug Report]
E.g. radosgw charm fails, when self-signed SSL certificate has IP address only (not hostname based).

2019-02-06 13:05:46 DEBUG identity-service-relation-changed Traceback (most recent call last):
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/identity-service-relation-changed", line 400, in <module>
2019-02-06 13:05:46 DEBUG identity-service-relation-changed hooks.execute(sys.argv)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2019-02-06 13:05:46 DEBUG identity-service-relation-changed self._hooks[hook_name]()
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1891, in wrapped_f
2019-02-06 13:05:46 DEBUG identity-service-relation-changed restart_functions)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2019-02-06 13:05:46 DEBUG identity-service-relation-changed r = lambda_f()
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1890, in <lambda>
2019-02-06 13:05:46 DEBUG identity-service-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/identity-service-relation-changed", line 245, in identity_changed
2019-02-06 13:05:46 DEBUG identity-service-relation-changed configure_https()
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/identity-service-relation-changed", line 389, in configure_https
2019-02-06 13:05:46 DEBUG identity-service-relation-changed setup_keystone_certs(CONFIGS)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 356, in _inner2_defer_if_unavailable
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 496, in setup_keystone_certs
2019-02-06 13:05:46 DEBUG identity-service-relation-changed get_ks_ca_cert(ksclient, auth_endpoint, certs_path)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 356, in _inner2_defer_if_unavailable
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 414, in get_ks_ca_cert
2019-02-06 13:05:46 DEBUG identity-service-relation-changed ca_cert = get_ks_cert(ksclient, auth_endpoint, 'ca')
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 356, in _inner2_defer_if_unavailable
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return f(*args, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-radosgw-int-0/charm/hooks/utils.py", line 384, in get_ks_cert
2019-02-06 13:05:46 DEBUG identity-service-relation-changed cert = ksclient.certificates.get_ca_certificate()
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/certificates.py", line 29, in get_ca_certificate
2019-02-06 13:05:46 DEBUG identity-service-relation-changed resp, body = self._client.get('/certificates/ca', authenticated=False)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return self.request(url, 'GET', **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2019-02-06 13:05:46 DEBUG identity-service-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return self.session.request(url, method, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2019-02-06 13:05:46 DEBUG identity-service-relation-changed return func(*args, **kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 405, in request
2019-02-06 13:05:46 DEBUG identity-service-relation-changed resp = send(**kwargs)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 443, in _send_request
2019-02-06 13:05:46 DEBUG identity-service-relation-changed raise exceptions.SSLError(msg)
2019-02-06 13:05:46 DEBUG identity-service-relation-changed keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://100.86.0.2:35357/v2.0/certificates/ca: hostname '100.86.0.2' doesn't
match '100.86.0.2'

[Gábor Mészáros (gabor.meszaros)]

works with python-urllib3 1.13.1-2ubuntu0.16.04.1~cloud0
but fails with 1.13.1-2ubuntu0.16.04.2~cloud0

Change introduced: https://bugs.launchpad.net/ubuntu/+source/python-urllib3/+bug/1771988

code segment that silently fails:
 96 def match_hostname(cert, hostname):
 97 """Verify that *cert* (in decoded format as returned by
 98 SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 and RFC 6125
 99 rules are followed, but IP addresses are not accepted for *hostname*.
100
101 CertificateError is raised on failure. On success, the function
102 returns nothing.
103 """
104 if not cert:
105 raise ValueError("empty or no certificate, match_hostname needs a "
106 "SSL socket or SSL context with either "
107 "CERT_OPTIONAL or CERT_REQUIRED")
108 try:
109 # Divergence from upstream: ipaddress can't handle byte str
110 host_ip = ipaddress.ip_address(_to_unicode(hostname))
111 except ValueError:
112 # Not an IP address (common case)
113 host_ip = None
114 except UnicodeError:
115 # Divergence from upstream: Have to deal with ipaddress not taking
116 # byte strings. addresses should be all ascii, so we consider it not
117 # an ipaddress in this case
118 host_ip = None
119 except AttributeError: <<<<<< throws AttributeError, because ipaddress is not available >>>>>>
120 # Divergence from upstream: Make ipaddress library optional
121 if ipaddress is None:
122 host_ip = None
123 else:
124 raise

from here:
  9 # ipaddress has been backported to 2.6+ in pypi. If it is installed on the
 10 # system, use it to handle IPAddress ServerAltnames (this was added in
 11 # python-3.5) otherwise only do DNS matching. This allows
 12 # backports.ssl_match_hostname to continue to be used all the way back to
 13 # python-2.4.
 14 try:
 15 import ipaddress
 16 except ImportError:
 17 ipaddress = None
 18

Simply installing python-ipaddress solves the issue

[Ante Karamatić (ivoks)]

When Trusty is installed, it comes with Trusty's version of python-urllib3, which does not recommend or depend on python-ipaddress.

Once charm is installed with cloud archive, it upgrades python-urllib3 to newer version. This version does recommend python-ipaddress, but upgrades do not install recommended packages. Therefore python-ipaddress is not installed.

Either package should make sure it depends on ipaddress or charms should install ipaddress as part of the process.

[James Page (james-page)]

"Either package should make sure it depends on ipaddress or charms should install ipaddress as part of the process."

as charms are not involved in normal day to day package upgrades it would seem appropriate to switch the recommends to a depends on the package to ensure its installed.

[James Page (james-page)]

Hmm that said, IP address SAN's did not work before so I'm not sure this should be a package fix - it is an optional depends.

[James Page (james-page)]

@Gabor - what's the impact of this issue? just the radosgw charm or is there a wider issue with communication between units?

[James Page (james-page)]

Sorry I missed some of the detail on #2 (about the upgrade from trusty->UCA version of urllib3).

[Ante Karamatić (ivoks)]

All OpenStack services are impacted.

[Gábor Mészáros (gabor.meszaros)]

keystone, and all that is charm related with it, as Ante said.

[James Page (james-page)]

I'm not even seeing a recommends on python-urllib3 in trusty/mitaka:

Package: python-urllib3
Priority: optional
Section: python
Installed-Size: 287
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Version: 1.13.1-2ubuntu0.16.04.2~cloud0
Recommends: ca-certificates, python-ndg-httpsclient, python-openssl, python-pyasn1

[Gábor Mészáros (gabor.meszaros)]

James, what more logs or details do you need, to accept this is a valid issue?

[James Page (james-page)]

Recommends are install when packages are upgraded:

# apt install python-urllib3
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  python-ipaddress python-pyasn1
Suggested packages:
  doc-base python-ntlm
Recommended packages:
  python-ndg-httpsclient
The following NEW packages will be installed:
  python-ipaddress python-pyasn1
The following packages will be upgraded:
  python-urllib3
1 upgraded, 2 newly installed, 0 to remove and 9 not upgraded.
Need to get 131 kB of archives.
After this operation, 596 kB of additional disk space will be used.
Do you want to continue? [Y/n]

So the right fix here is to update python-urllib3 to have a Recommends on python-ipaddress.

[Gábor Mészáros (gabor.meszaros)]

James, I can confirm that the fix is working properly on our sites.

[James Page (james-page)]

<jamespage> James Page rbasak: the recommends does not exist in xenial BUT python-ipaddress is already installed on a xenial base image I think
12:11 <rbalint> Balint Reczey cjwatson, ok, sorry for the wrong vote
12:11 <jamespage> James Page so a happy co-incidence means that stuff works OK on xenial, but not on trusty, where python-ipaddress is not installed
12:12 rbasak: I'm actually OK with marking the xenial task as invalid - I fixed the UCA part with a UCA specific patch which we auto-apply to add the recommends
12:12 <rbasak> Robie Basak jamespage: OK - but it's currently in Xenial unapproved and not in Trusty unapproved.
12:13 <jamespage> James Page rbasak: that's true - but this is a UCA specific issue on trusty, not a general trusty issue
12:13 ricab → ricab|lunch
12:13 <rbasak> Robie Basak I see, OK.
12:13 <jamespage> James Page rbasak: xenial feeds trusty/mitaka in the UCA
12:13 <rbasak> Robie Basak ah
12:14 I think I understand the situation then, thanks.
12:14 Are we in agreement to drop the Xenial SRU then? I'm still open to it if you want to push for it; I would prefer not to do it though.
12:16 <jamespage> James Page rbasak: updated bug