[Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA

[Bas de Bruijne]

[edit]: from comment #2:

======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======

---

The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.

In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:

================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/77 active idle 10.246.164.228 Filebeat ready.
  hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
  landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
  logrotated/77 active idle 10.246.164.228 Unit is ready.
  nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
  octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
  octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
  prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
  public-policy-routing/48 active idle 10.246.164.228 Unit is ready
  telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
  ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
  filebeat/78 active idle 10.246.167.116 Filebeat ready.
  hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
  landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
  logrotated/78 active idle 10.246.167.116 Unit is ready.
  nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
  octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
  octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
  prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
  public-policy-routing/49 active idle 10.246.167.116 Unit is ready
  telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
  ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/75 active idle 10.246.167.69 Filebeat ready.
  hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
  landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
  logrotated/75 active idle 10.246.167.69 Unit is ready.
  nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
  octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
  octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
  prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
  public-policy-routing/46 active idle 10.246.167.69 Unit is ready
  telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
  ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================

In the debug log, we see the following message:
================
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
    _render(os.path.basename(conf))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
    content = template.render(context)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/openstack_https_frontend.conf", line 1, in top-level template code
    {% if options.endpoints -%}
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
    return getattr(obj, attribute)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
    int_port = ch_cluster.determine_api_port(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
    if https():
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
    cert_utils.get_requests_for_local_unit("certificates")
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
    'ca': data['ca'],
KeyError: 'ca'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
    octavia_charm.render_with_interfaces(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
    self.render_configs(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
    _render('_'.join(conf.split(os.path.sep))[1:])
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
    raise e
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
    template = template_env.get_template(source)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
    return self._load_template(name, globals)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
    raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================

It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html

[Alex Kavanagh (ajkavanagh)]

Bas, please include in the bug description all of the information indicated in https://docs.openstack.org/charm-guide/yoga/community/software-bug.html

e.g. the essential information:

an overview of your environment

a detailed description of the problem and how it differs from your expected outcome

specific software versions in use

Ubuntu

OpenStack

Juju (see Juju versions)

MAAS

Even better is to include in the title the OpenStack version and the Ubuntu series. For any charms being mentioned the juju channel (and series if different from the main model).

Thanks.

[Bas de Bruijne (basdbruijne)]

An overview of the versions used can be found in the first link, but for completeness:

======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======

Our environment is using a persistent Maas 3.2.8 deployment with a juju 2.9.44 controller bootstrapped and an OpenStack bundle (https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/generated/generated/openstack/bundle.yaml) deployed.

The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.

[Alex Kavanagh (ajkavanagh)]

> An overview of the versions used can be found in the first link ...

Just as a minor comment, but if this information isn't included in the description, then every person who reads the bug has to go and find that information (and know that it's in the first link). By including it, and a commentary on what is expected, what is wrong, etc. this really helps in comprehension of the bug, understanding which versions maybe affected and provides really important context. Thank you for the additional information.

[Bas de Bruijne (basdbruijne)]

We have 5 more occurrences today, I wonder if this is a regression in the Octavia charm revision 158 (released 2 days ago). I am looking at commit [1] in particular as a possible cause, though I would have expected that to be present in other charms as well.

[1] https://github.com/juju/charm-helpers/commit/58097330bd15cfd7b539481c55718c6cee8ca124

[Alex Kavanagh (ajkavanagh)]

@bas; yup, good sleuthing! It's a very subtle error, and I don't think the logic is quite correct. Obviously, there is an error in assuming that the 'ca' key is appearing in the data sent from the certificate provider if it's a legacy request. Unpicking this might be a bit complex due to two different methods of requested certificates on the relation.

[Alex Kavanagh (ajkavanagh)]

I think this may be related to the vault caching issue as the show-unit for octavia/0 is

     vault/0:
        in-scope: true
        data:
          ca: |-
            -----BEGIN CERTIFICATE-----
            MIID7TCCAtWgAwIBAgIUfk0uomNXrMmdlGzOur44HWh/v5gwDQYJKoZIhvcNAQEL
            ...
            lots of data ...
            ...
            /3VHMeHB6G8kcfYZtdq5LeLtrC0eRstDY6uXlLz1afXWAwqxwbtt9FlRNttK2ZNh
            dnueH3hzZlhhITaN2Tsg+GDYb0McdBTPEDUeeaVP2R5LZZl+OnG7OA==
            -----END RSA PRIVATE KEY-----
          private-address: 10.246.168.166
      vault/1:
        in-scope: true
        data:
          egress-subnets: 10.246.168.167/32
          ingress-address: 10.246.168.167
          private-address: 10.246.168.167
      vault/2:
        in-scope: true
        data:
          egress-subnets: 10.246.168.168/32
          ingress-address: 10.246.168.168
          private-address: 10.246.168.168

The code in question (from charm-helpers/charmhelpers/contrib/openstack
/cert_utils.py) is:

def get_requests_for_local_unit(relation_name=None):
    """Extract any certificates data targeted at this unit down relation_name.

    :param relation_name: str Name of relation to check for data.
    :returns: List of bundles of certificates.
    :rtype: List of dicts
    """
    local_name = local_unit().replace('/', '_')
    raw_certs_key = '{}.processed_requests'.format(local_name)
    relation_name = relation_name or 'certificates'
    bundles = []
    for rid in relation_ids(relation_name):
        sent = relation_get(rid=rid, unit=local_unit())
        legacy_keys = ['certificate_name', 'common_name']
        is_legacy_request = set(sent).intersection(legacy_keys)
        for unit in related_units(rid):
            data = relation_get(rid=rid, unit=unit)
            if data.get(raw_certs_key):
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': json.loads(data[raw_certs_key])})
            elif is_legacy_request:
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': {sent['common_name']:
                              {'cert': data.get(local_name + '.server.cert'),
                               'key': data.get(local_name + '.server.key')}}})

But the main issue is, I believe, a timing. Octavia requests a certificate using the old form, but it has not yet been processed. i.e. raw_certs_key == {localname}processed_requests is not present in the data bag from vault, but the two keys 'certificate_name' and 'common_name' are present in the outbound relation from octavia. This results in the "elif is_legacy_request" being true, and thus the erroneous access to data['ca'] which isn't available in the data bag.

And essentially, it's a timing issue. Vault hasn't been initialised, so it's not sending any certs, but due to how hooks are handled in reactive charms, it's still running the handler render which goes off to look at all the relations for the context.

Fixing this may be as simple as just checking that 'ca' is available, and if not, bailing o...

[Alex Kavanagh (ajkavanagh)]

charm-helpers fix: https://github.com/juju/charm-helpers/pull/824

Will need backporting once it is merged.

[Alex Kavanagh (ajkavanagh)]

Related fix that probably caused this issue: https://bugs.launchpad.net/charm-ironic-api/+bug/2022980

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/890556

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/890565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/890556
Committed: https://opendev.org/openstack/charm-ironic-api/commit/113ccde9d445bedec482efdec12841aa32ecc91a
Submitter: "Zuul (22348)"
Branch: master

commit 113ccde9d445bedec482efdec12841aa32ecc91a
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: I3d454dbddda8006178797d381cd07856faca897d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/890565
Committed: https://opendev.org/openstack/charm-octavia/commit/fc85965b3bb9601a4b273b07c6d55bcbd5917620
Submitter: "Zuul (22348)"
Branch: master

commit fc85965b3bb9601a4b273b07c6d55bcbd5917620
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: Id3636d55f26d05ffed994e3ccc0575ec66a5e389

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891496

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891546

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891716

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891860

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891890

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891899

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891958

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891996

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892031

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891546
Committed: https://opendev.org/openstack/charm-octavia/commit/ee1259750892536e1206ce182836c5983b861836
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ee1259750892536e1206ce182836c5983b861836
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: I0e7e2e67109f39a507b69de9d4f1e6abccf32f58

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537
Committed: https://opendev.org/openstack/charm-ironic-api/commit/2a4af9398493be9d11d8dbe0751e51aa0fb39556
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2a4af9398493be9d11d8dbe0751e51aa0fb39556
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: Ie91b970c7d8233bcd27e3ebaa5804e23ab40e443

[Alan Baghumian (alanbach)]

I'm also encountering this exact same issue today, this fix has not yet been released for Yoga/Stable unfortunately.

[Felipe Reyes (freyes)]

On Thu, 2023-08-31 at 18:25 +0000, Alan Baghumian wrote:
> I'm also encountering this exact same issue today, this fix has not yet
> been released for Yoga/Stable unfortunately.

that's correct and in sync with the tasks this bug has, you can monitor this patch too -
https://review.opendev.org/c/openstack/charm-octavia/+/891496

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967
Reason: Abandoning in favour of https://review.opendev.org/c/openstack/charm-octavia/+/892336 which is also a rebuild request, and thus essentially duplicates the rebuild/push.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022
Committed: https://opendev.org/openstack/charm-ironic-api/commit/f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: Ic9c479d1076b22419706dfa8fb7940033d13d46e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487
Committed: https://opendev.org/openstack/charm-ironic-api/commit/dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I214d62984423c50e4c32b051e64ab5b2e851b9a8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707
Committed: https://opendev.org/openstack/charm-ironic-api/commit/8389fc923e01931b72dbddbe661f4c76e78e76b1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 8389fc923e01931b72dbddbe661f4c76e78e76b1
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: If06ad8d782a47edaa320da8d710211f3ec04e28c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851
Committed: https://opendev.org/openstack/charm-ironic-api/commit/7330d5ae7f78b3bdb94c5b72480c06958c534798
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7330d5ae7f78b3bdb94c5b72480c06958c534798
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: I3c7ca65fca93a43d659b3c4cd48ae2ef7dc35c79

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/892031
Committed: https://opendev.org/openstack/charm-octavia/commit/dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: I51b9784be407f724515aed655f4325be40057871

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891496
Committed: https://opendev.org/openstack/charm-octavia/commit/57561cff59b6155709a791237ad885b01bc2fb20
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 57561cff59b6155709a791237ad885b01bc2fb20
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I41ea5a58e0391d398fab2ecd44999e6f32a4054b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891716
Committed: https://opendev.org/openstack/charm-octavia/commit/aab20025c9219a76ceebcccbd04a481aeaeff54e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit aab20025c9219a76ceebcccbd04a481aeaeff54e
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: I1f7368e2851402d67f71f69590d1be6c8ddd25e3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891860
Committed: https://opendev.org/openstack/charm-octavia/commit/5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: Id883860c21bff001e693760a0535fd427e9c6d18

[Alan Baghumian (alanbach)]

Finally got tired of waiting and manually copied over the charmhelpers/contrib/openstack/cert_utils.py file to my Octavia units and performed a juju resolve. They went past the error and all green now!

Hopefully the updated charm will be available since this is not a proper way of fixing things :-D

$ sudo cp -v cert_utils.py /var/lib/juju/agents/unit-octavia-<unit#>/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/cert_utils.py

Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 17:56:44-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
octavia/17* active idle 9/lxd/12 10.1.15.145 9876/tcp Unit is ready
  lds-client-focal/691 active idle 10.1.15.145 System successfully registered
  octavia-hacluster/99 active idle 10.1.15.145 Unit is ready and clustered
  octavia-mysql-router/122* active idle 10.1.15.145 Unit is ready
  octavia-ovn-chassis/119* active idle 10.1.15.145 Unit is ready
octavia/18 active idle 102/lxd/5 10.1.15.146 9876/tcp Unit is ready
  lds-client-focal/690 active idle 10.1.15.146 System successfully registered
  octavia-hacluster/98 active idle 10.1.15.146 Unit is ready and clustered
  octavia-mysql-router/123 active idle 10.1.15.146 Unit is ready
  octavia-ovn-chassis/118 active idle 10.1.15.146 Unit is ready
octavia/19 active idle 104/lxd/4 10.1.15.147 9876/tcp Unit is ready
  lds-client-focal/689 active idle 10.1.15.147 System successfully registered
  octavia-hacluster/97* active idle 10.1.15.147 Unit is ready and clustered
  octavia-mysql-router/121 active idle 10.1.15.147 Unit is ready
  octavia-ovn-chassis/117 active idle 10.1.15.147 Unit is ready

Machine State Address Inst id Series AZ Message
9 started 10.1.8.12 os-vm-2 focal default Deployed
9/lxd/12 started 10.1.15.145 juju-b096f0-9-lxd-12 focal default Container started
102 started 10.1.8.25 ...

[Alex Kavanagh (ajkavanagh)]

Hi @alanback

I'm not sure which channel of the octavia-charm you are using, but the fix has landed to all of the following channels: ussuri, wallaby, xena, yoga, zed, 2023.1 (antelope) and master. The victoria and (bionic - queens, rocky, stein, train) are currently not building.

There's a tooling issue, as the patches have landed but the bug was not updated. I've manually updated the ones that have landed now.

Sorry for the incorrect information.

Thanks!

[Alan Baghumian (alanbach)]

Hi Alex,

I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when it cam out a couple of weeks ago:

channels: |
  yoga/stable: 20bbe01 2023-09-08 (193) 41MB

$ juju status octavia
Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 09:22:04-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

However that did not fix this issue until I manually copied the file yesterday.

So, if 193 is indeed the latest for Yoga/Stable, it did not fix the problem.

Hope this helps!

Best,
Alan

[Felipe Reyes (freyes)]

On Sun, 2023-09-24 at 16:24 +0000, Alan Baghumian wrote:
> Hi Alex,
>
> I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when
> it cam out a couple of weeks ago:[...]
when you upgraded, were the hooks in error state?, if they were, you need to use the "--force-units"
flag to make juju upgrade the charm, otherwise it will wait for the hook failures get resolved.

You can check the output of "juju show-status-log <entity name>" to know when/if the upgrade-charm
hook was executed.

[Alex Kavanagh (ajkavanagh)]

Hi; you made me check! :)

I downloaded the charm manually for octavia (juju download octavia --channel=yoga/stable) and checked the charmhelpers/contrib/cert_utils.py at line 417 and it does contain the fix.

Felipe raises a point on whether the units were actually upgraded rather than still waiting on failed hooks. Another method would be to add a unit (if possible) and see if it works okay.

[Alan Baghumian (alanbach)]

Hi @Felipe! You nailed it:

$ juju show-status-log octavia/19 --days 30 | grep -c upgrade-charm
0

The hook never actually ran even after running juju resolve on units.

Hi @Alex! Sorry I made you double check this and thanks much for checking.

I am now running the upgrade hook.

Best,
Alan

[Alan Baghumian (alanbach)]

We are good! Just refreshed to xena/stable and then back to yoga/stable. All good.

$ juju refresh octavia --channel xena/stable

$ juju refresh octavia --channel yoga/stable

Best,
Alan

[Felipe Reyes (freyes)]

On Tue, 2023-09-26 at 16:51 +0000, Alan Baghumian wrote:
> We are good! Just refreshed to xena/stable and then back to yoga/stable.
> All good.

that's great.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892005
Reason: Train is about to transition to End of Life. Open patches needs to be abandoned before branch deletion.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891996
Reason: Train is about to transition to End of Life. Open patches needs to be abandoned before branch deletion.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891996
Committed: https://opendev.org/openstack/charm-ironic-api/commit/0314051cb222e48628ca064e576141941ae884b6
Submitter: "Zuul (22348)"
Branch: stable/train

commit 0314051cb222e48628ca064e576141941ae884b6
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 15:04:43 2023 +0100

    [train] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/832
    Closes-Bug: #2028683

    Change-Id: Iebe98a4b7793cce9f7e8266a2efe218f46a77862

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/892005
Committed: https://opendev.org/openstack/charm-octavia/commit/4c08e572130a69990e900b7ee2070341463bbb48
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4c08e572130a69990e900b7ee2070341463bbb48
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 15:04:43 2023 +0100

    [train] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/832
    Closes-Bug: #2028683

    Change-Id: I6619dc5ca6e6a0d4123f2958560e8a3a98c89446