Comment 0 for bug 2028683

In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:

================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/77 active idle 10.246.164.228 Filebeat ready.
  hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
  landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
  logrotated/77 active idle 10.246.164.228 Unit is ready.
  nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
  octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
  octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
  prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
  public-policy-routing/48 active idle 10.246.164.228 Unit is ready
  telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
  ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
  filebeat/78 active idle 10.246.167.116 Filebeat ready.
  hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
  landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
  logrotated/78 active idle 10.246.167.116 Unit is ready.
  nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
  octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
  octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
  prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
  public-policy-routing/49 active idle 10.246.167.116 Unit is ready
  telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
  ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/75 active idle 10.246.167.69 Filebeat ready.
  hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
  landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
  logrotated/75 active idle 10.246.167.69 Unit is ready.
  nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
  octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
  octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
  prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
  public-policy-routing/46 active idle 10.246.167.69 Unit is ready
  telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
  ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================

In the debug log, we see the following message:
================
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
    _render(os.path.basename(conf))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
    content = template.render(context)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/openstack_https_frontend.conf", line 1, in top-level template code
    {% if options.endpoints -%}
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
    return getattr(obj, attribute)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
    int_port = ch_cluster.determine_api_port(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
    if https():
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
    cert_utils.get_requests_for_local_unit("certificates")
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
    'ca': data['ca'],
KeyError: 'ca'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
    octavia_charm.render_with_interfaces(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
    self.render_configs(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
    _render('_'.join(conf.split(os.path.sep))[1:])
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
    raise e
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
    template = template_env.get_template(source)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
    return self._load_template(name, globals)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
    raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================

It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html