| 2023-07-25 15:27:14 |
Bas de Bruijne |
bug |
|
|
added bug |
| 2023-07-25 15:27:38 |
Bas de Bruijne |
tags |
|
cdo-qa foundations-engine |
|
| 2023-07-25 17:54:00 |
Alex Kavanagh |
charm-octavia: status |
New |
Incomplete |
|
| 2023-07-25 18:11:10 |
Bas de Bruijne |
summary |
Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA |
[Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA |
|
| 2023-07-25 18:11:14 |
Bas de Bruijne |
charm-octavia: status |
Incomplete |
New |
|
| 2023-07-25 20:01:45 |
Alex Kavanagh |
description |
In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:
================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/77 active idle 10.246.164.228 Filebeat ready.
hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
logrotated/77 active idle 10.246.164.228 Unit is ready.
nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
public-policy-routing/48 active idle 10.246.164.228 Unit is ready
telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
filebeat/78 active idle 10.246.167.116 Filebeat ready.
hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
logrotated/78 active idle 10.246.167.116 Unit is ready.
nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
public-policy-routing/49 active idle 10.246.167.116 Unit is ready
telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/75 active idle 10.246.167.69 Filebeat ready.
hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
logrotated/75 active idle 10.246.167.69 Unit is ready.
nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
public-policy-routing/46 active idle 10.246.167.69 Unit is ready
telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================
In the debug log, we see the following message:
================
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
_render(os.path.basename(conf))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
content = template.render(context)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
raise rewrite_traceback_stack(source=source)
File "templates/openstack_https_frontend.conf", line 1, in top-level template code
{% if options.endpoints -%}
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
return getattr(obj, attribute)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
int_port = ch_cluster.determine_api_port(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
if https():
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
cert_utils.get_requests_for_local_unit("certificates")
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
'ca': data['ca'],
KeyError: 'ca'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
octavia_charm.render_with_interfaces(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
self.render_configs(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
_render('_'.join(conf.split(os.path.sep))[1:])
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
raise e
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
template = template_env.get_template(source)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
return self._load_template(name, globals)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================
It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html |
[edit]: from comment #2:
======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======
---
In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:
================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/77 active idle 10.246.164.228 Filebeat ready.
hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
logrotated/77 active idle 10.246.164.228 Unit is ready.
nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
public-policy-routing/48 active idle 10.246.164.228 Unit is ready
telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
filebeat/78 active idle 10.246.167.116 Filebeat ready.
hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
logrotated/78 active idle 10.246.167.116 Unit is ready.
nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
public-policy-routing/49 active idle 10.246.167.116 Unit is ready
telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/75 active idle 10.246.167.69 Filebeat ready.
hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
logrotated/75 active idle 10.246.167.69 Unit is ready.
nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
public-policy-routing/46 active idle 10.246.167.69 Unit is ready
telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================
In the debug log, we see the following message:
================
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
_render(os.path.basename(conf))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
content = template.render(context)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
raise rewrite_traceback_stack(source=source)
File "templates/openstack_https_frontend.conf", line 1, in top-level template code
{% if options.endpoints -%}
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
return getattr(obj, attribute)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
int_port = ch_cluster.determine_api_port(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
if https():
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
cert_utils.get_requests_for_local_unit("certificates")
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
'ca': data['ca'],
KeyError: 'ca'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
octavia_charm.render_with_interfaces(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
self.render_configs(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
_render('_'.join(conf.split(os.path.sep))[1:])
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
raise e
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
template = template_env.get_template(source)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
return self._load_template(name, globals)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================
It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html |
|
| 2023-07-25 20:10:02 |
Alex Kavanagh |
description |
[edit]: from comment #2:
======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======
---
In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:
================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/77 active idle 10.246.164.228 Filebeat ready.
hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
logrotated/77 active idle 10.246.164.228 Unit is ready.
nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
public-policy-routing/48 active idle 10.246.164.228 Unit is ready
telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
filebeat/78 active idle 10.246.167.116 Filebeat ready.
hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
logrotated/78 active idle 10.246.167.116 Unit is ready.
nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
public-policy-routing/49 active idle 10.246.167.116 Unit is ready
telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/75 active idle 10.246.167.69 Filebeat ready.
hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
logrotated/75 active idle 10.246.167.69 Unit is ready.
nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
public-policy-routing/46 active idle 10.246.167.69 Unit is ready
telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================
In the debug log, we see the following message:
================
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
_render(os.path.basename(conf))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
content = template.render(context)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
raise rewrite_traceback_stack(source=source)
File "templates/openstack_https_frontend.conf", line 1, in top-level template code
{% if options.endpoints -%}
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
return getattr(obj, attribute)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
int_port = ch_cluster.determine_api_port(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
if https():
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
cert_utils.get_requests_for_local_unit("certificates")
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
'ca': data['ca'],
KeyError: 'ca'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
octavia_charm.render_with_interfaces(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
self.render_configs(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
_render('_'.join(conf.split(os.path.sep))[1:])
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
raise e
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
template = template_env.get_template(source)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
return self._load_template(name, globals)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================
It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html |
[edit]: from comment #2:
======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======
---
The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.
In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:
================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/77 active idle 10.246.164.228 Filebeat ready.
hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
logrotated/77 active idle 10.246.164.228 Unit is ready.
nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
public-policy-routing/48 active idle 10.246.164.228 Unit is ready
telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
filebeat/78 active idle 10.246.167.116 Filebeat ready.
hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
logrotated/78 active idle 10.246.167.116 Unit is ready.
nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
public-policy-routing/49 active idle 10.246.167.116 Unit is ready
telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
filebeat/75 active idle 10.246.167.69 Filebeat ready.
hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
logrotated/75 active idle 10.246.167.69 Unit is ready.
nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
public-policy-routing/46 active idle 10.246.167.69 Unit is ready
telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================
In the debug log, we see the following message:
================
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
_render(os.path.basename(conf))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
content = template.render(context)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
raise rewrite_traceback_stack(source=source)
File "templates/openstack_https_frontend.conf", line 1, in top-level template code
{% if options.endpoints -%}
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
return getattr(obj, attribute)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
int_port = ch_cluster.determine_api_port(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
if https():
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
cert_utils.get_requests_for_local_unit("certificates")
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
'ca': data['ca'],
KeyError: 'ca'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
octavia_charm.render_with_interfaces(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
self.render_configs(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
_render('_'.join(conf.split(os.path.sep))[1:])
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
charmhelpers.core.templating.render(
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
raise e
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
template = template_env.get_template(source)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
return self._load_template(name, globals)
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================
It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html |
|
| 2023-07-26 17:55:22 |
Alex Kavanagh |
charm-octavia: status |
New |
Triaged |
|
| 2023-07-26 17:55:24 |
Alex Kavanagh |
charm-octavia: importance |
Undecided |
Critical |
|
| 2023-07-26 19:11:11 |
Alex Kavanagh |
charm-octavia: assignee |
|
Alex Kavanagh (ajkavanagh) |
|
| 2023-07-27 05:26:45 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2023-07-27 12:02:29 |
Alex Kavanagh |
charm-octavia: status |
Triaged |
In Progress |
|
| 2023-07-27 13:27:18 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api |
|
| 2023-07-27 13:27:41 |
Alex Kavanagh |
bug task added |
|
charm-helpers |
|
| 2023-07-27 13:27:47 |
Alex Kavanagh |
charm-helpers: status |
New |
In Progress |
|
| 2023-07-27 13:27:51 |
Alex Kavanagh |
charm-helpers: importance |
Undecided |
Critical |
|
| 2023-07-27 13:27:54 |
Alex Kavanagh |
charm-helpers: assignee |
|
Alex Kavanagh (ajkavanagh) |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/victoria |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/victoria |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/yoga |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/yoga |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/2023.1 |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/2023.1 |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/xena |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/xena |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/ussuri |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/ussuri |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/trunk |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/trunk |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/wallaby |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/wallaby |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
nominated for series |
|
charm-octavia/zed |
|
| 2023-07-27 13:29:06 |
Alex Kavanagh |
bug task added |
|
charm-octavia/zed |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/victoria |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/victoria |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/2023.1 |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/2023.1 |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/yoga |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/yoga |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/ussuri |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/ussuri |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/xena |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/xena |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/trunk |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/trunk |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/zed |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/zed |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
nominated for series |
|
charm-ironic-api/wallaby |
|
| 2023-07-27 13:29:33 |
Alex Kavanagh |
bug task added |
|
charm-ironic-api/wallaby |
|
| 2023-08-04 17:47:35 |
OpenStack Infra |
charm-ironic-api: status |
New |
In Progress |
|
| 2023-08-07 22:03:03 |
OpenStack Infra |
charm-ironic-api: status |
In Progress |
Fix Committed |
|
| 2023-08-07 22:04:19 |
OpenStack Infra |
charm-octavia: status |
In Progress |
Fix Committed |
|
| 2023-08-14 08:50:41 |
Yoshi Kadokawa |
bug |
|
|
added subscriber Yoshi Kadokawa |
| 2023-08-15 14:16:19 |
OpenStack Infra |
charm-ironic-api/yoga: status |
New |
In Progress |
|
| 2023-08-15 21:24:20 |
OpenStack Infra |
charm-ironic-api/2023.1: status |
New |
In Progress |
|
| 2023-08-17 12:58:06 |
OpenStack Infra |
charm-ironic-api/xena: status |
New |
In Progress |
|
| 2023-08-17 15:31:18 |
OpenStack Infra |
charm-ironic-api/wallaby: status |
New |
In Progress |
|
| 2023-08-17 19:45:10 |
OpenStack Infra |
charm-ironic-api/victoria: status |
New |
In Progress |
|
| 2023-08-18 11:16:11 |
OpenStack Infra |
charm-ironic-api/ussuri: status |
New |
In Progress |
|
| 2023-08-18 16:41:11 |
OpenStack Infra |
charm-ironic-api/zed: status |
New |
In Progress |
|
| 2023-08-21 16:49:11 |
Jeffrey Chang |
bug |
|
|
added subscriber Jeffrey Chang |
| 2023-08-22 18:01:58 |
OpenStack Infra |
charm-ironic-api/2023.1: status |
In Progress |
Fix Committed |
|
| 2023-09-08 15:48:03 |
OpenStack Infra |
charm-ironic-api/zed: status |
In Progress |
Fix Committed |
|
| 2023-09-08 15:48:08 |
OpenStack Infra |
charm-ironic-api/yoga: status |
In Progress |
Fix Committed |
|
| 2023-09-08 15:48:14 |
OpenStack Infra |
charm-ironic-api/xena: status |
In Progress |
Fix Committed |
|
| 2023-09-08 15:48:19 |
OpenStack Infra |
charm-ironic-api/wallaby: status |
In Progress |
Fix Committed |
|
| 2023-09-24 00:45:33 |
Alan Baghumian |
bug |
|
|
added subscriber Alan Baghumian |
| 2023-09-24 16:01:21 |
Alex Kavanagh |
charm-octavia/ussuri: status |
New |
Fix Released |
|
| 2023-09-24 16:01:33 |
Alex Kavanagh |
charm-octavia/wallaby: status |
New |
Fix Released |
|
| 2023-09-24 16:01:43 |
Alex Kavanagh |
charm-octavia/xena: status |
New |
Fix Released |
|
| 2023-09-24 16:01:53 |
Alex Kavanagh |
charm-octavia/yoga: status |
New |
Fix Released |
|
| 2023-09-24 16:02:04 |
Alex Kavanagh |
charm-octavia/zed: status |
New |
Fix Released |
|
| 2023-09-24 16:02:17 |
Alex Kavanagh |
charm-octavia/2023.1: status |
New |
Fix Released |
|
| 2023-09-24 16:02:38 |
Alex Kavanagh |
charm-helpers: status |
In Progress |
Fix Released |
|
| 2023-09-24 16:03:34 |
Alex Kavanagh |
charm-octavia/victoria: status |
New |
In Progress |
|
| 2023-12-23 16:11:34 |
OpenStack Infra |
tags |
cdo-qa foundations-engine |
cdo-qa foundations-engine in-stable-train |
|
