charm documentation unclear about usage of the ssl_* configuration options

[José Pekkarinen]

As per charm documentation, implementation of tls endpoint for keystone requires to add
a base64 encoded copy of your self-signed certificate. After deploying the charm, following
the documentation, the status message of the charm renders as:

Services not running that should be: apache2

Digging into the unit I see the following:

root@juju-8fb8f4-1-lxd-0:~# cat /etc/apache2/ssl/keystone/cert_keystone.mydomain.com
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVLVENDQXhHZ0F3SUJBZ0lKQUtvaFNFcmtI
...
LS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

So charm is doing a json dump forgetting that it requires to decode the base64.

Thanks!

José.

[Frode Nordahl (fnordahl)]

What does your ssl_ca, ssl_cert and ssl_key configuration options look like? I would suspect they contain strings that have been base64 encoded twice.

[Stuart Bishop (stub)]

Consider sniffing the value, decoding if base64, and removing the unnecessary requirement of base64 encoding certificates.

[José Pekkarinen (koalinux)]

A bit more info on the case. I have an overlay for ssl in fce where the file looks
like this:

variables:
  ssl_ca: &ssl_ca include-base64://certs/ca-b64.crt
  ssl_cert: &ssl_cert include-base64://certs/dev-b64.crt
  ssl_key: &ssl_key include-base64://certs/dev-b64.key
applications:
  cinder:
    options:
      ssl_ca: *ssl_ca
      ssl_cert: *ssl_cert
      ssl_key: *ssl_key
  glance:
    options:
      ssl_ca: *ssl_ca
      ssl_cert: *ssl_cert
      ssl_key: *ssl_key
...

The overlay is added in master.yaml, as:

- name: openstack
  type: openstack
  parent: juju_maas_controller
  config:
    juju_model: openstack
    bundles: # Primary bundle followed by optional overlays.
    - bundle.yaml
    - overlay-hostnames.yaml
    - overlay-ssl.yaml

which effectively executes:

juju deploy --debug --overlay config/overlay-hostnames.yaml --overlay config/overlay-ssl.yaml config/bundle.yaml

The content of the certificate files is like:

$ cat fce/config/certs/ca-b64.crt
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVKakNDQXc2Z0F3SUJBZ0lKQVBzem5zYW5x
...
RCBDRVJUSUZJQ0FURS0tLS0tCg==

$ cat fce/config/certs/dev-b64.crt
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVLVENDQXhHZ0F3SUJBZ0lKQUtvaFNFcmtI
...
LS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

$ cat fce/config/certs/dev-b64.key
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQkFBS0NBUUVBdnpoK3A2TEph
...
LUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

So this, as per the keystone charm documentation may render
in having a regular certificate under /etc/apache2/ssl/keystone,
instead I have the base64 encoded version of it, so at some point
it has been encoded twice.

I tried, to put the certificate directly, no encoding, python
will backtrace, complaining about padding, being a red herring.
I also tried to put the base64 version, dropping the include-base64,
and changing it for include directly, to ensure python yaml is not
encoding it again. The only thing that mitigate the issue for me
was described in this PR:

https://github.com/juju/charm-helpers/pull/366

If there is any further info you would need for this to clearly
identify it's a bug and it needs attention, please ask about it.

[Frode Nordahl (fnordahl)]

As you display in comment #3 the file on-disk referenced in the bundle contain already base64 encoded data.

The bundle you show use the ``include-base64`` bundle keyword. As per juju implementation [0] the ``include-base64`` keyword reads the data from disk and base64 __ENCODES__ it.

Subsequently you have exactly what I said in comment #1, the certificate data is Base64 encoded twice.

As you would expect the charm does exactly as you have asked it to, it base64 decodes the data (once) and writes your data which is base64 encoded twice to disk.

You must either update your bundle to use the ``include`` keyword or update the on-disk secrets to not be base64 encoded when you feed it to juju with the ``include-base64`` bundle keyword.

0: https://github.com/juju/juju/commit/ec584b10f71ae0cc4a4a9d936d15f8b7875c4d2b

[José Pekkarinen (koalinux)]

keystone configuration as an example:

variables:

  openstack-origin: &openstack-origin |
    deb http://cloud-archive.dev.mydomain.com/ubuntu bionic-updates/stein main
    |
    -----BEGIN PGP PUBLIC KEY BLOCK-----

    mQINBFAqSlgBEADPKwXUwqbgoDYgR20zFypxSZlSbrttOKVPEMb0HSUx9Wj8VvNC
    ...
    -----END PGP PUBLIC KEY BLOCK-----

  oam-space: &oam-space oam-space
  # This is OpenStack Admin network; for adminURL endpoints
  admin-space: &admin-space o7k-internal-space
  # This is OpenStack Public network; for publicURL endpoints
  public-space: &public-space oam-space
  # This is OpenStack Internal network; for internalURL endpoints
  internal-space: &internal-space o7k-internal-space

  worker-multiplier: &worker-multiplier 0.25
  keystone-vip: &keystone-vip "12.16.2.5"
  openstack-region: &openstack-region RegionOne

  keystone:
    charm: cs:keystone
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      vip: *keystone-vip
      region: *openstack-region
      preferred-api-version: 3
    to:
    - lxd:1
    - lxd:2
    - lxd:3

[José Pekkarinen (koalinux)]

Sample of final status after deploy:

$ juju status keystone
Model Controller Cloud/Region Version SLA Timestamp
openstack foundations-maas maas_cloud 2.6.6 unsupported 12:54:40Z

App Version Status Scale Charm Store Rev OS Notes
hacluster-keystone waiting 3 hacluster jujucharms 60 ubuntu
keystone 15.0.0 blocked 3 keystone jujucharms 306 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* blocked executing 1/lxd/0 12.16.2.9 5000/tcp Services not running that should be: apache2
  hacluster-keystone/0* active idle 12.16.2.9 Unit is ready and clustered
keystone/1 blocked executing 2/lxd/0 12.16.2.7 5000/tcp Services not running that should be: apache2
  hacluster-keystone/2 waiting executing 12.16.2.7 Resource: res_ks_b4e9d20_vip not yet configured
keystone/2 blocked executing 3/lxd/0 12.16.2.5 5000/tcp Services not running that should be: apache2
  hacluster-keystone/1 active executing 12.16.2.5 Unit is ready and clustered

$ juju ssh keystone/0
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-58-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

 System information disabled due to load higher than 4.0

10 packages can be updated.
0 updates are security updates.

Last login: Thu Aug 29 11:57:43 2019 from 12.16.2.2
ubuntu@juju-620b84-1-lxd-0:~$ sudo -i
root@juju-620b84-1-lxd-0:~# cat /etc/apache2/ssl/keystone/cert_keystone.dev.mydomain.com
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVLVENDQXhHZ0F3SUJBZ0lKQUtvaFNFcmtI
...
LS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

[Frode Nordahl (fnordahl)]

I would be interested in your response to comment 4 and any results pertaining to the two suggestions to you there.

[José Pekkarinen (koalinux)]

Frode, you are right with it, the thing is that if I read the following
paragraph from keystone charm documentation, I do understand it has to
be encoded in advance:

keystone:
  charm: cs:keystone
  num_units: 1
  options:
    ssl_ca: include-base64://path-to-base64-encoded-ca-data
    ssl_cert: include-base64://path-to-base64-encoded-certificate-data
    ssl_key: include-base64://path-to-base64-encoded-key-data

So the problem here is the documentation leading to confusion.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679371
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=28711f99513649eac0de0663c1ee08e63a1e23c3
Submitter: Zuul
Branch: master

commit 28711f99513649eac0de0663c1ee08e63a1e23c3
Author: Frode Nordahl <email address hidden>
Date: Fri Aug 30 05:01:26 2019 +0200

    Fix misleading example for certificate configuration

    Change-Id: I45edd7dc14d240e56c52d74147142f30f8c592d0
    Closes-Bug: #1841429