Wrong symlink is created for certificate received via Vault relation

[Vladimir Grevtsev]

charm: cs:openstack-dashboard-276

Problem statement: Certificates are being issued for both FQDN and MAAS-created hostname - AND the second one becoming default one as symlink points on it: https://pastebin.canonical.com/p/5wFPCzh2Kw/ . That leads to situation, when clients are receiving wrong cert while trying to reach service using expected FQDN: https://pastebin.canonical.com/p/yzcw2zTb3C/

openstack-dashboard unit logs: https://drive.google.com/file/d/1ZBl9L788FeFzGy9OC0DjdsliOjexOwj0/view?usp=sharing
relation metadata: https://pastebin.canonical.com/p/zvBCNfHWjB/
unit config: https://pastebin.canonical.com/p/Y9PMxr4mZ3/

bundle part:

applications:

...

openstack-dashboard:
    charm: cs:openstack-dashboard
    num_units: 3
    constraints: *oam-space-constr
    bindings:
      "": *public-space
      shared-db: *internal-space
    options:
      openstack-origin: *openstack-origin
      webroot: "/"
      secret: "xxxxxxxxxxxxxxxxxx"
      vip: *dashboard-vip
      neutron-network-l3ha: true
      neutron-network-lb: true
      neutron-network-firewall: fal
      cinder-backup: false
      password-retrieve: true
      endpoint-type: publicURL

...

relations:
 - ["openstack-dashboard:certificates", "vault:certificates"]

[Vladimir Grevtsev (vlgrevtsev)]

Subscribed field-high since this affects one already Bootstack-live environment + one being deployed right now.

[Vladimir Grevtsev (vlgrevtsev)]

The only w/a for this right now is to overwrite symlinks - after that customer should receive valid response:

$ juju run --application openstack-dashboard 'ln -sf /etc/apache2/ssl/horizon/key_dashboard.openstack.local /etc/apache2/ssl/horizon/key_dashboard; ln -sf /etc/apache2/ssl/horizon/cert_dashboard.openstack.local /etc/apache2/ssl/horizon/cert_dashboard; service apache2 restart'
- Stdout: ""
  UnitId: openstack-dashboard/0
- Stdout: ""
  UnitId: openstack-dashboard/1
- Stdout: ""
  UnitId: openstack-dashboard/2

$ curl -k https://dashboard.openstack.local -v 2>&1 | grep "CN"
* subject: CN=dashboard.openstack.local
* issuer: CN=Vault Intermediate Certificate Authority (charm-pki-local)

[Frode Nordahl (fnordahl)]

When setting the ``os-public-hostname`` configuration option on the ``openstack-dashboard`` charm you indeed get two certificates installed but the certificate representing the configured hostname is not selected.

I would agree this is unexpected behavior.

[Frode Nordahl (fnordahl)]

Other classic charms appear to deal with this correctly by using this template for the Apache config: https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/templates/openstack_https_frontend

[Frode Nordahl (fnordahl)]

The context ``openstack-dashboard`` charm uses for ApacheSSL is not able to cope with that though.

A short term fix would be a kludge for the case of presence of ``os-public-hostname`` configuration option in the charm.

Long term fix would be to ditch the custom ApacheSSL context in favor of the standard shared one in ``charm-helpers`` along with a new Apache configuration template that meets Horizon's needs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/638443

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/638478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638443
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=256f971c78574f017eb802ab7097231698b96ae0
Submitter: Zuul
Branch: master

commit 256f971c78574f017eb802ab7097231698b96ae0
Author: Frode Nordahl <email address hidden>
Date: Thu Feb 21 16:38:55 2019 +0100

    Use correct certificate when ``os-public-hostname`` configration option is set

    Note that this is a short term kludge/fix, on the long term
    we should ditch the charm specific ApacheSSLContext and use
    the common one from charm-helpers with an adapted Apache
    config inspired from the ``openstack_https_fronted`` template

    Change-Id: I74c17113f431c4c21f638be6abffaeeb693f1462
    Closes-Bug: #1816621

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638603

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638478
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=d52307da7c164274d53259417909b146ac256d45
Submitter: Zuul
Branch: stable/18.11

commit d52307da7c164274d53259417909b146ac256d45
Author: Frode Nordahl <email address hidden>
Date: Thu Feb 21 16:38:55 2019 +0100

    Use correct certificate when ``os-public-hostname`` configration option is set

    Note that this is a short term fix, on the long term
    we should replace the charm specific ApacheSSLContext with
    the common one from charm-helpers with an adapted Apache
    config inspired from the ``openstack_https_fronted`` template.

    Change-Id: I74c17113f431c4c21f638be6abffaeeb693f1462
    Closes-Bug: #1816621
    (cherry picked from 256f971c78574f017eb802ab7097231698b96ae0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638603
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=19915f6806a053bf8c25cc589b5964058dc7bfe2
Submitter: Zuul
Branch: master

commit 19915f6806a053bf8c25cc589b5964058dc7bfe2
Author: Frode Nordahl <email address hidden>
Date: Fri Feb 22 08:50:41 2019 +0100

    Use common ApacheSSLContext

    Remove the custom ApacheSSLContext class and use the common
    one from ``charmhelpers.contrib.openstack`` instead.

    Update ``default-ssl`` template so we can make use of multiple
    endpoints with SNI.

    Sync required changes to charm-helpers.

    Change-Id: Icc990448d2c7469c5253d04ad43371d01d5580d9
    Related-Bug: #1816621

[Vladimir Grevtsev (vlgrevtsev)]

cs:openstack-dashboard-277 (after recent backport to stable):

openstack-dashboard/0 error idle 4/lxd/3 xxx.xxx.xxx.xxx 80/tcp,443/tcp hook failed: "certificates-relation-changed"

unit logs: https://pastebin.canonical.com/p/TJ6n4RYVhT/

steps to reproduce:

1) deploy bundle with dashboard, vault and relation between them
2) unseal Vault and authorize them

e.g this is reproduced even without passing via get-csr/upload-signed-csr routine.

[Vladimir Grevtsev (vlgrevtsev)]

This is what happened inside openstack-dashboard container:

ubuntu@juju-b5d11d-4-lxd-3:~$ ls -lah /etc/apache2/ssl/horizon/
total 16K
dr-xr-xr-x 2 root root 4.0K Feb 22 14:11 .
dr-xr-xr-x 3 root root 4.0K Feb 22 14:11 ..
lrwxrwxrwx 1 root root 72 Feb 22 14:11 cert_dashboard -> /etc/apache2/ssl/horizon/cert_dashboard.openstack.local
lrwxrwxrwx 1 root root 71 Feb 22 14:11 key_dashboard -> /etc/apache2/ssl/horizon/key_dashboard.openstack.local

[Frode Nordahl (fnordahl)]

Interesting, the temporary fix that got to stable/18.11 was tested successfully in a vault environment with ``os-public-hostname`` configuration option set.

We have implemented the proper fix @ master in https://review.openstack.org/638603

Could you help verify if this works for you at master with cs:~openstack-charmers-next/openstack-dashboard ?

[Vladimir Grevtsev (vlgrevtsev)]

cs:~openstack-charmers-next/openstack-dashboard-399 not working: https://pastebin.canonical.com/p/5XJgmwvvRW/

openstack-dashboard/1 error idle 5/lxd/5 x.x.x.x 80/tcp,443/tcp hook failed: "certificates-relation-changed"