Charm removes new lines and spaces from certificates

[Nicolas Pochet]

In order to troubleshoot why the charm was not discovering Octavia, I added some print() to the charm:

diff --git a/lib/charms/layer/openstack.py b/lib/charms/layer/openstack.py
index 4752426..e0a267c 100644
--- a/lib/charms/layer/openstack.py
+++ b/lib/charms/layer/openstack.py
@@ -67,6 +67,7 @@ def get_credentials():
                                     stdout=subprocess.PIPE,
                                     stderr=subprocess.PIPE)
             _creds_data = yaml.safe_load(result.stdout.decode('utf8'))
+ print(_creds_data)
             _merge_if_set(creds_data, _normalize_creds(_creds_data))
         except FileNotFoundError:
             pass # juju trust not available
@@ -92,6 +93,7 @@ def get_credentials():
             status.blocked(str(e))
             return False

+ print(creds_data)
     if all(creds_data[k] for k in required_fields):
         _save_creds(creds_data)
         return True

We can then see in the debug-log that the charm is stripping all new lines and spaces:

unit-openstack-integrator-0: 08:09:59 DEBUG unit.openstack-integrator/0.upgrade-charm {'type': 'openstack', 'name': 'openstack_cloud', 'region': 'Area51', 'endpoint': 'https
://keystone.customer:5000/v3', 'identityendpoint': '', 'storageendpoint': '', 'credential': {'authtype': 'userpass', 'attributes': {'domain-name': '
', 'password': 'Meh', 'project-domain-name': 'admin_domain', 'tenant-name': 'admin', 'user-domain-name': 'admin_domain', 'username': 'admin'}, 'redacted': []},
'cacertificates': ['-----BEGIN CERTIFICATE-----\nMIIDITCCAgmgAwIBAgIUeQxHSsZt6auk1oW+SRFXC4T6nNcwDQYJKoZIhvcNAQEL\nBQAwIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElubWFyc2F0MB4XDTE5MT
EwNDE1\nMTQzOFoXDTI5MTEwMTE1MTQzOFowIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElu\nbWFyc2F0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA09qCmv8jF+N1\ndl/ae3VQV95FG7WFrjS6fbZ1TpXkO9VsP
KhA9lRUBxs58noKIkMIUeXYy4wvSu28\nX67NqB2bv3iyns/mEzPYE1GxtFXIPhkKO22vqVLZ0CFAuV47AhqDOXtyqwwfxoBT\nKxMi430UCb+3cPaev/mZMlvf6iJfdihyPfMEwtIanS/QKgEvykhP1kAZ36ActFmK\nWnJtjBBF
UKQIBQzguMTqUXX7wvwRegK8lgXiZ6iZiOza0C7hSdBVylcKeaqoLnP5\nW93m3YZTXc08A30PieTJQFD6Bm+41Kv2FxQAXjRnCzvIJL44zJXjLmnUdZbSzdl8\nPpu3wJu9cQIDAQABo1MwUTAdBgNVHQ4EFgQUwQsYIyqud2WQk
AlcDwIuu7nAvnYw\nHwYDVR0jBBgwFoAUwQsYIyqud2WQkAlcDwIuu7nAvnYwDwYDVR0TAQH/BAUwAwEB\n/zANBgkqhkiG9w0BAQsFAAOCAQEAn5oQYeyaxcqOjzUxbkEy4pOJMg/nTKkt+8yh\nFSqUv1Vc3HGg65uGq08eJDq9
AP7PrfvSQJWQpFBS80bNN8idCmhMutpA8X6+Z0wv\n0p5dzQFAUdSLLN0so4iXKtPk5wp0r84W0xbqWPRWRSw+lCe1WrMK+ARDpPv+AxOW\nf7JFQkqzEsWu6RCjy0KobOy7PPq17wXEhXynNcMAXjQe9DkTBb34K6PYku1Ftxfr\
n3IRWaSrDB9BJTje6/tmz7IcO8ss+Y3gUZeaqTLdZz8RJUlJqNqfdTQif2hKLYjro\nBwZYRQo8TkDmSlz00LwQSo1xLX27nGHB621pgNCZbJMKvZOrQg==\n-----END CERTIFICATE-----\n']}
unit-openstack-integrator-0: 08:09:59 DEBUG unit.openstack-integrator/0.upgrade-charm {'auth_url': 'https://keystone.customer:5000/v3', 'region': 'A
rea51', 'username': 'admin', 'password': 'Meh', 'user_domain_name': 'admin_domain', 'project_domain_name': 'admin_domain', 'project_name': 'admin', 'endpoint_tl
s_ca': 'BEGINCERTIFICATEMIIDITCCAgmgAwIBAgIUeQxHSsZt6auk1oW+SRFXC4T6nNcwDQYJKoZIhvcNAQELBQAwIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElubWFyc2F0MB4XDTE5MTEwNDE1MTQzOFoXDTI5MTEwMTE1M
TQzOFowIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElubWFyc2F0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA09qCmv8jF+N1dl/ae3VQV95FG7WFrjS6fbZ1TpXkO9VsPKhA9lRUBxs58noKIkMIUeXYy4wvSu28X6
7NqB2bv3iyns/mEzPYE1GxtFXIPhkKO22vqVLZ0CFAuV47AhqDOXtyqwwfxoBTKxMi430UCb+3cPaev/mZMlvf6iJfdihyPfMEwtIanS/QKgEvykhP1kAZ36ActFmKWnJtjBBFUKQIBQzguMTqUXX7wvwRegK8lgXiZ6iZiOza0C7
hSdBVylcKeaqoLnP5W93m3YZTXc08A30PieTJQFD6Bm+41Kv2FxQAXjRnCzvIJL44zJXjLmnUdZbSzdl8Ppu3wJu9cQIDAQABo1MwUTAdBgNVHQ4EFgQUwQsYIyqud2WQkAlcDwIuu7nAvnYwHwYDVR0jBBgwFoAUwQsYIyqud2WQ
kAlcDwIuu7nAvnYwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAn5oQYeyaxcqOjzUxbkEy4pOJMg/nTKkt+8yhFSqUv1Vc3HGg65uGq08eJDq9AP7PrfvSQJWQpFBS80bNN8idCmhMutpA8X6+Z0wv0p5dzQFAU
dSLLN0so4iXKtPk5wp0r84W0xbqWPRWRSw+lCe1WrMK+ARDpPv+AxOWf7JFQkqzEsWu6RCjy0KobOy7PPq17wXEhXynNcMAXjQe9DkTBb34K6PYku1Ftxfr3IRWaSrDB9BJTje6/tmz7IcO8ss+Y3gUZeaqTLdZz8RJUlJqNqfdTQ
if2hKLYjroBwZYRQo8TkDmSlz00LwQSo1xLX27nGHB621pgNCZbJMKvZOrQg==', 'version': '3'}

We can see, in the unit that this malformed certificate is written to disk:
ubuntu@juju-2f1f32-kubernetes-15:~$ cat /etc/openstack-integrator/ca.crt
BEGINCERTIFICATEMIIDITCCAgmgAwIBAgIUeQxHSsZt6auk1oW+SRFXC4T6nNcwDQYJKoZIhvcNAQELBQAwIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElubWFyc2F0MB4XDTE5MTEwNDE1MTQzOFoXDTI5MTEwMTE1MTQzOFowIDELMAkGA1UEBhMCVUsxETAPBgNVBAoMCElubWFyc2F0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA09qCmv8jF+N1dl/ae3VQV95FG7WFrjS6fbZ1TpXkO9VsPKhA9lRUBxs58noKIkMIUeXYy4wvSu28X67NqB2bv3iyns/mEzPYE1GxtFXIPhkKO22vqVLZ0CFAuV47AhqDOXtyqwwfxoBTKxMi430UCb+3cPaev/mZMlvf6iJfdihyPfMEwtIanS/QKgEvykhP1kAZ36ActFmKWnJtjBBFUKQIBQzguMTqUXX7wvwRegK8lgXiZ6iZiOza0C7hSdBVylcKeaqoLnP5W93m3YZTXc08A30PieTJQFD6Bm+41Kv2FxQAXjRnCzvIJL44zJXjLmnUdZbSzdl8Ppu3wJu9cQIDAQABo1MwUTAdBgNVHQ4EFgQUwQsYIyqud2WQkAlcDwIuu7nAvnYwHwYDVR0jBBgwFoAUwQsYIyqud2WQkAlcDwIuu7nAvnYwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAn5oQYeyaxcqOjzUxbkEy4pOJMg/nTKkt+8yhFSqUv1Vc3HGg65uGq08eJDq9AP7PrfvSQJWQpFBS80bNN8idCmhMutpA8X6+Z0wv0p5dzQFAUdSLLN0so4iXKtPk5wp0r84W0xbqWPRWRSw+lCe1WrMK+ARDpPv+AxOWf7JFQkqzEsWu6RCjy0KobOy7PPq17wXEhXynNcMAXjQe9DkTBb34K6PYku1Ftxfr3IRWaSrDB9BJTje6/tmz7IcO8ss+Y3gUZeaqTLdZz8RJUlJqNqfdTQif2hKLYjroBwZYRQo8TkDmSlz00LwQSo1xLX27nGHB621pgNCZbJMKvZOrQg==

We can also see, in the openstack-cloud-controller-manager logs that the certificate is not OK:

kubectl logs openstack-cloud-controller-manager-bbtsb -n kube-system

...
F1118 08:14:50.385121 1 controllermanager.go:121] Cloud provider could not be initialized: could not init cloud provider "openstack": error reading /etc/config/endpoint-ca.cert: data does not contain any valid RSA or ECDSA certificates

[Nicolas Pochet (npochet)]

The issue seems to be that without checking if the certificate that we receive is really base64 encoded, we try to decode it: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/lib/charms/layer/openstack.py#L208-L211

Trying that in a python interpreter:
https://pastebin.ubuntu.com/p/gkv3ZYVSG5/
We can see that we lost all spaces and new lines and that the last output is not even a base64 encoded file...

[Nicolas Pochet (npochet)]

Pull request that fixes this bug: https://github.com/juju-solutions/charm-openstack-integrator/pull/24

[Nicolas Pochet (npochet)]

Subscribing field-medium as we have a work-around and a PR to fix this issue.

[Nobuto Murata (nobuto)]

Subscribing ~field-high as it's blocking one of the ongoing deployments.

A possible workaround would be not using Juju's "trust" feature at all, which means the documented functionality is unusable. It looks like the charm assumes base64 encoded CA certificate passed to the charm option, but when using `juju trust` to inherit those info from the controller it must be a plain text.

[output of `credential-get` from a unit]
https://pastebin.canonical.com/p/SjMtWSGpqy/

The b64decode will be skipped with some CA certs because of "binascii.Error: Incorrect padding". But we cannot recreate CA certs as those are given, and cannot rely on a lucky always.
https://pastebin.canonical.com/p/W2S7jXGNdF/

[Nobuto Murata (nobuto)]

The actual trace from the debug-log of the charm fwiw.

unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: Invoking reactive handler: reactive/openstack.py:64:handle_requests
unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: status-set: maintenance: Granting integration requests
unit-openstack-integrator-0: 00:50:37 ERROR unit.openstack-integrator/0.juju-log clients:35: Error while trying to detect Octavia
Traceback (most recent call last):
  File "lib/charms/layer/openstack.py", line 116, in detect_octavia
    catalog = {s['Name'] for s in _openstack('catalog', 'list')}
  File "lib/charms/layer/openstack.py", line 273, in _openstack
    output = _run_with_creds('openstack', *args, '--format=yaml')
  File "lib/charms/layer/openstack.py", line 260, in _run_with_creds
    CA_CERT_FILE.write_text(ca_cert.decode('utf8') + '\n')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 2: invalid start byte

unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: status-set: maintenance: Granting request for kubernetes-master/0
unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: Finished request for kubernetes-master/0
unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: status-set: maintenance: Granting request for kubernetes-master/1
unit-openstack-integrator-0: 00:50:37 INFO unit.openstack-integrator/0.juju-log clients:35: Finished request for kubernetes-master/1

[Wouter van Bommel (woutervb)]

Applying changes from pull request https://github.com/juju-solutions/charm-openstack-integrator/pull/24 on a local version of the charm, and using this patched version allowed us to overcome the issues with the deployment.
But this breaks management, as an upgrade will potentially break it (again)