Charm Helpers

  1. Bug #2012801
  2. Comment #0

Comment 0 for bug 2012801

Revision history for this message
Nobuto Murata (nobuto) wrote : Services not running that should be: apache2, SSLCertificateFile: file '/etc/apache2/ssl/*/cert_* does not exist or is empty

maas: 1:3.3.1-13169-g.94920eb1e-0ubuntu1~22.04.1
juju: 2.9.42-ubuntu-amd64
charm-keystone: lastest/edge 9bdc837
charm-vault: latest/edge d8f0840

There are multiple reasons ending up with "Services not running that should be: apache2". However, this bug report focuses on the following condition:
- MAAS provider
- OpenStack API services are deployed in LXD containers on top of bare metal
- one certificate is written as /etc/apache2/ssl/*/cert_<fqdn>
- symlink creation fails from /etc/apache2/ssl/*/cert_<vip> to /etc/apache2/ssl/*/cert_<hostname>
- apache2 fails to start because of missing /etc/apache2/ssl/*/cert_<vip>

> $ sudo systemctl status apache2
> × apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
> Active: failed (Result: exit-code) since Sat 2023-03-25 12:50:59 UTC; 13h ago
> Docs: https://httpd.apache.org/docs/2.4/
> CPU: 53ms
>
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Starting The Apache HTTP Server...
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43820]: SSLCertificateFile: file '/etc/apache2/ssl/keystone/cert_192.168.151.99' does not exist or is empty
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: Action 'start' failed.
> Mar 25 12:50:59 juju-043209-2-lxd-0 apachectl[43817]: The Apache error log may have more information.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: apache2.service: Failed with result 'exit-code'.
> Mar 25 12:50:59 juju-043209-2-lxd-0 systemd[1]: Failed to start The Apache HTTP Server.

Long story short, this issue happens when responses of reverse DNS lookup to an IP address are inconsistent in get_hostname().
https://github.com/juju/charm-helpers/blob/6e302bab63e22e356d77d76a5c6d90d9d24c6390/charmhelpers/contrib/network/ip.py#L497

In this case, keystone charm uses the initial get_request() call to request a certificate and write the cert based on the output. Then, the charm uses the second get_request() call to get a path to create a symlink to then ends up with no such file.
https://github.com/juju/charm-helpers/blob/b5725ac546372e7d4004d15095f79cdd5e7da687/charmhelpers/contrib/openstack/cert_utils.py#L105

[requested and written cert]
/etc/apache2/ssl/keystone/cert_eth0.juju-043209-2-lxd-0.maas (w/ eth0.)
-> exists

[the patch trying to create a symlink to]
/etc/apache2/ssl/keystone/cert_juju-043209-2-lxd-0.maas (w/o eth0.)
-> does not exist

unit-keystone-2: 12:45:41 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'eth0.juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'eth0.juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': '{"eth0.juju-043209-2-lxd-0.maas": {"sans": ["192.168.151.131", "192.168.151.99"]}}', 'unit_name': 'keystone_2'}
unit-keystone-2: 12:50:54 WARNING unit.keystone/2.juju-log certificates:10: get_request: self.hostname_entry={'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']},self.entries=[{'cn': 'juju-043209-2-lxd-0.maas', 'addresses': ['192.168.151.131', '192.168.151.99']}],sans=['192.168.151.131', '192.168.151.99'],request={'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}},req={'cert_requests': {'juju-043209-2-lxd-0.maas': {'sans': ['192.168.151.131', '192.168.151.99']}}, 'unit_name': 'keystone_2'}

This is due to how MAAS DNS works.

$ dig +short @192.168.151.1 -x 192.168.151.131
eth0.juju-043209-2-lxd-0.maas.
juju-043209-2-lxd-0.maas.

$ grep -I -C1 -r juju-043209-2-lxd-0 /var/lib/bind/maas/
/var/lib/bind/maas/zone.maas-$ORIGIN maas.
/var/lib/bind/maas/zone.maas:juju-043209-2-lxd-0 A 192.168.151.131
/var/lib/bind/maas/zone.maas:$ORIGIN juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.maas-eth0 A 192.168.151.131
--
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-129 PTR juju-fe03b8-2-lxd-8.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa:131 PTR eth0.juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa: PTR juju-043209-2-lxd-0.maas.
/var/lib/bind/maas/zone.151.168.192.in-addr.arpa-134 PTR eth0.juju-043209-0-lxd-0.maas.

>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'eth0.juju-043209-2-lxd-0.maas.'
>>> str(dns.resolver.query(dns.reversename.from_address("192.168.151.131"), "PTR")[0])
'juju-043209-2-lxd-0.maas.'

How to reproduce:

1. prepare MAAS provider for Juju
2. prepare 3 machines for workload (enlisting VMs as if bare metal or using Pod VMs are fine)
3. deploy a test bundle
4. unlock vault
5. repeat deployment and destroy-model until "Services not running that should be: apache2" shows up in juju status

The step 3&4 can be unattended as follows.
====
juju destroy-model keystone-test --no-wait --force -y; \
    juju add-model keystone-test maas && \
    juju deploy ./keystone-ha_vault_edge.yaml && time juju-wait -w --exclude vault

VAULT_ADDR="http://$(juju run --unit vault/leader -- network-get certificates --ingress-address):8200"
export VAULT_ADDR

vault_init_output="$(vault operator init -key-shares=1 -key-threshold=1 -format json)"
vault operator unseal "$(echo "$vault_init_output" | jq -r .unseal_keys_b64[])"

VAULT_TOKEN="$(echo "$vault_init_output" | jq -r .root_token)"
export VAULT_TOKEN

juju run-action --wait vault/leader authorize-charm \
 token="$(vault token create -ttl=10m -format json | jq -r .auth.client_token)"
juju run-action vault/leader --wait generate-root-ca
====