apache2 fails to start due to missing certificate and key

[David Ames]

Apache attempts to start with SSL configured.

root@juju-34081f-6-lxd-3:/etc/apache2/sites-enabled# cat openstack_https_frontend.conf
Listen 8031
<VirtualHost 10.244.40.247:8031>
    ServerName gnocchi-internal.maas
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/gnocchi/cert_gnocchi-internal.maas
    SSLCertificateKeyFile /etc/apache2/ssl/gnocchi/key_gnocchi-internal.maas
    ProxyPass / http://localhost:8021/
    ProxyPassReverse / http://localhost:8021/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 10.244.40.247:8031>
    ServerName gnocchi-public.maas
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/gnocchi/cert_gnocchi-public.maas
    SSLCertificateKeyFile /etc/apache2/ssl/gnocchi/key_gnocchi-public.maas
    ProxyPass / http://localhost:8021/
    ProxyPassReverse / http://localhost:8021/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

However the certificate and key do not exist.

root@juju-34081f-6-lxd-3:/etc/apache2/sites-enabled# cat /etc/apache2/ssl/gnocchi/key_gnocchi-public.maas
cat: /etc/apache2/ssl/gnocchi/key_gnocchi-public.maas: No such file or directory
root@juju-34081f-6-lxd-3:/etc/apache2/sites-enabled# cat /etc/apache2/ssl/gnocchi/cert_gnocchi-public.maas
cat: /etc/apache2/ssl/gnocchi/cert_gnocchi-public.maas: No such file or directory

The output of juju status will show

        workload-status:
          current: blocked
          message: 'Services not running that should be: apache2'
          since: 03 Mar 2018 12:12:33Z

For all the gnocchi and aodh units.

[Jason Hobbs (jason-hobbs)]

Subscribed to Field Critical SLA, as this bug will cause all SSL enabled deployments of gnocchi to fail.

[Ashley Lai (alai)]

The issue can be reproduced with the following deployment. Crashdump is attached.

juju status: https://pastebin.canonical.com/p/MmR9nRdsrz/
bundle: https://pastebin.canonical.com/p/rC4yyKhW6j/

overlay-hostnames.yaml: https://pastebin.canonical.com/p/fszx8HTGQ6/
overlay-ssl.yaml: https://pastebin.canonical.com/p/YptYYc7XDw/
overlay-mysql-noha.yaml: https://pastebin.canonical.com/p/938fM5xn6q/

deploy command:
juju deploy -m foundations-maas:admin/ssl ./bundle.yaml --overlay ./overlay-hostnames.yaml --overlay ./overlay-ssl.yaml --overlay ./overlay-mysql-noha.yaml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/548067

[David Ames (thedac)]

Ashley,

Can you please do a test run with cs:~thedac/gnocchi-1? I think this resolves the Services not running apache2 problem. We may find other issues, but I am interested if we get past that one.

[David Ames (thedac)]

Performed some live testing during deploy time that highlights the race condition.

After, the first gnocchi unit advertises itself as ready to the ceilometer the other units are still in progress. The SSL version of the test shows this quite clearly. Hitting the VIP which then gets sent to HAProxy and an indeterminate gnocchi back end, some back ends are ready and respond and some have not even setup SSL yet:

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:46:47-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 144 [application/json]
Unable to establish SSL connection.

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:03-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
Saving to: ‘STDOUT’

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:22-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:24-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
Saving to: ‘STDOUT’

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:44-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 144 [application/json]
Saving to: ‘STDOUT’

2018-02-26 10:49:46 (11.4 MB/s) - written to stdout [144/144]

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:48-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to esta...

[Ashley Lai (alai)]

@David - does cs:~thedac/gnocchi-1 includes the fix for 1749280 ?

[David Ames (thedac)]

Ashley, it has the updates to help mitigate, yes. It is built on the following:

https://review.openstack.org/#/c/545491/
https://review.openstack.org/#/c/548067/

[Ashley Lai (alai)]

The charm is applied, we will get some runs with the updated charm tonight.

[Liam Young (gnuoy)]

I think the bug could be generalised to "API service advertises endpoint before all units of service are ready" and it applies to most (all ?) of our api charms I expect. So, I think option 2 is probably the correct one but I would say option 1 is fine for this particular instance. I am not keen on option 3. I think the contract between charms should be such that if a charm advertises and endpoint it is declaring that that endpoint is ready to use.

As I say, option 2 probably needs to be done regardless of whether option 1 is implemented. thedac has already done a lot of work in various charms to delay endpoint advertising. I think that work needs to be applied here with the additional delay of "If ssl is enabled don't advertise
endpoint until certs/keys are configured and installed on all units".

[James Page (james-page)]

I'd agree with Liam's analysis - if we want to get a release out in the short-term, we'll need to go with 1) for now and work towards 2) over the next cycle; its a common pattern so needs careful design and implementation so not one to try and squeeze in after feature freeze.

[Ashley Lai (alai)]

We see the same error with gnocchi charm pointed to cs:~thedac/gnocchi-1. bundle.yaml and crashdump are attached.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/548067
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=b0a1551e56f2a540942cbd745fa4658057100c5e
Submitter: Zuul
Branch: master

commit b0a1551e56f2a540942cbd745fa4658057100c5e
Author: David Ames <email address hidden>
Date: Mon Feb 26 08:55:14 2018 -0800

    Explicitly call configure_ssl

    Although, layer_openstack_api calls configure_ssl the ordering is
    indeterminate. Explicitly call configure_ssl in the charm layer before
    services are enabled to avoid attempting to start apache2 before
    certificates are available.

    Change-Id: Icb40bbcf4ca920be202584e9e25ffd1c2b9c8d61
    Partial-Bug: #1750915

[Chris Gregan (cgregan)]

Still seeing this issue quite often even with fixes in place. This will continue to be a field blocker at this occurrence rate

[Jason Hobbs (jason-hobbs)]

attached a crashdump from our most recent reproduction.

bundle:
http://paste.ubuntu.com/p/hNfNH4xHhV/

[Ryan Beisner (1chb1n)]

The bundle in comment #15 is not a reproducer. It appears to be missing overlays as provided earlier. Please confirm if/whether those have changed or are the same, preferrably by providing the complete stack of bundles with each run. Thank you.

[Jason Hobbs (jason-hobbs)]

here are the requested overlays from that run: http://paste.ubuntu.com/p/ySBtQYX6xb/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550104
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=4949dc24127a0de34195841f855b909f3ccf13ca
Submitter: Zuul
Branch: master

commit 4949dc24127a0de34195841f855b909f3ccf13ca
Author: Liam Young <email address hidden>
Date: Tue Mar 6 14:54:50 2018 +0000

    Render certs for all endpoint types

    When using user supplied certs directly to the charms make sure
    that the bundles which are passed in are rendered for all
    supported endpoint types (internal, admin and public).

    Closes-Bug: #1750915
    Change-Id: If62fd5528470653586b2be3aaef7f829e5abad40

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/550120

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550120
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=e3f0707e6a5af8451a13b21f43ac50f361a042f9
Submitter: Zuul
Branch: master

commit e3f0707e6a5af8451a13b21f43ac50f361a042f9
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:27:46 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: Ia6f6ee555f611da552337826f6fc6c87e34c7e55
    Closes-Bug: #1750915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550113
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=ce1e84794af48eb65266ab316830c50b3e2a2f25
Submitter: Zuul
Branch: master

commit ce1e84794af48eb65266ab316830c50b3e2a2f25
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:25:43 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: I40385c436d928e996239053f9103dc66b38f3e0f
    Closes-Bug: #1750915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550117
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=49c5fdeb828dfba647375d9effc82b9a3ccfbae1
Submitter: Zuul
Branch: master

commit 49c5fdeb828dfba647375d9effc82b9a3ccfbae1
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:27:08 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: I287151ede2156b58b3057d8d569a3298acee8664
    Closes-Bug: #1750915

[Ashley Lai (alai)]

https://solutions.qa.canonical.com/#/qa/testRun/9b7e6a95-ee90-4096-8ed6-480de5ff68b4

The patch fixed the apache2 issue. ceilometer still failed with the following error.

2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 288, in get
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/client.py", line 35, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 192, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.session.request(url, method, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return wrapped(*args, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 578, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer auth_headers = self.get_auth_headers(auth)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 905, in get_auth_headers
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return auth.get_headers(self, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer token = self.get_token(session)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 89, in get_token
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.get_access(session).auth_token
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 135, in get_access
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 ...