ERROR octavia.controller.worker.v2.controller_worker jinja2.exceptions.TemplateNotFound: amphora_agent_conf.template

[Felipe Reyes]

[ Impact ]
Octavia fails to provision Amphora for load balancers.

[ Test Plan ]
Install Octavia as part of a Charmed OpenStack deployment for Caracal.
Create a loadbalancer - creation will fail with stack trace from original bug report.

[ Where problems could occur ]
The fix for this is to ensure that data files in the Python source tree are included in the package installation; this is done by providing a MANIFEST to ensure that this happens; side effects are unlikely.

[Original Bug Report]
[Impact]

Octavia fails to provision the amphora, the stacktrace logged is:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task
    result = task.execute(**arguments)
  File "/usr/lib/python3/dist-packages/octavia/controller/worker/v2/tasks/compute_tasks.py", line 199, in execute
    return super().execute(
  File "/usr/lib/python3/dist-packages/octavia/controller/worker/v2/tasks/compute_tasks.py", line 122, in execute
    agent_cfg = agent_jinja_cfg.AgentJinjaTemplater()
  File "/usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/agent_jinja_cfg.py", line 34, in __init__
    self.agent_template = jinja_env.get_template(
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 997, in get_template
    return self._load_template(name, globals)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 958, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 125, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 214, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: amphora_agent_conf.template

When searching for a package where this file is, apt-file can't find any. https://packages.ubuntu.com/search?searchon=contents&keywords=amphora_agent_conf.template&mode=exactfilename&suite=noble&arch=any

This file has been around for many years - https://opendev.org/openstack/octavia/commits/branch/master/octavia/amphorae/backends/agent/templates/amphora_agent_conf.template - and the code that's trying to use it hasn't really received changes during the Caracal cycle - https://opendev.org/openstack/octavia/commits/branch/master/octavia/amphorae/backends/agent/agent_jinja_cfg.py

[Environment]

* OpenStack 2024.1 (Caracal)
* python3-octavia 1:14.0.0-0ubuntu1~cloud0

[Felipe Reyes (freyes)]

Adding charm-octavia in case the charm is at fault, although everything seems to point out to a packaging issue.

[Felipe Reyes (freyes)]

The Debian package in sid has this file - https://packages.debian.org/search?searchon=contents&keywords=amphora_agent_conf.template&mode=path&suite=unstable&arch=any

File Packages
/usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/templates/amphora_agent_conf.template python3-octavia

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/923793

[Launchpad Janitor (janitor)]

This bug was fixed in the package octavia - 1:14.0.0-0ubuntu2

---------------
octavia (1:14.0.0-0ubuntu2) oracular; urgency=medium

  [ Felipe Reyes ]
  * d/patches/install-missing-files.patch: Ensure that all data files
    in the Python module directories are included in the binary package
    (LP: #2072526).

  [ James Page ]
  * d/control,d/p/tenacity-8.5-compat.patch: Fix compatibility with newer
    Tenacity versions.

 -- Felipe Reyes <email address hidden> Wed, 10 Jul 2024 07:07:44 +0100

[Andreas Hasenack (ahasenack)]

Please include in the test plan a comparison of installed files by the package before the fix, and after the fix, to be sure we are not suddenly including something we shouldn't.

[Andreas Hasenack (ahasenack)]

Hello Felipe, or anyone else affected,

Accepted octavia into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/octavia/1:14.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Hello Felipe, or anyone else affected,

Accepted octavia into caracal-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:caracal-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-caracal-needed to verification-caracal-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-caracal-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Felipe Reyes (freyes)]

The verification has been successful for jammy-caracal, here it's the relevant chunk of the testing log:

**withtout** proposed:

-06 04:13:29 [INFO] ======================================================================
2024-08-06 04:13:29 [INFO] ERROR: test_create_loadbalancer (zaza.openstack.charm_tests.octavia.tests.LBAASv2Test)
2024-08-06 04:13:29 [INFO] Create load balancer.
2024-08-06 04:13:29 [INFO] ----------------------------------------------------------------------
2024-08-06 04:13:29 [INFO] Traceback (most recent call last):
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/octavia/tests.py", line 502, in test_create_loadbalancer
2024-08-06 04:13:29 [INFO] lb = self._create_lb_resources(self.octavia_client,
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/octavia/tests.py", line 368, in _create_lb_resources
2024-08-06 04:13:29 [INFO] resp = self.wait_for_lb_resource(
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 289, in wrapped_f
2024-08-06 04:13:29 [INFO] return self(f, *args, **kw)
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in __call__
2024-08-06 04:13:29 [INFO] do = self.iter(retry_state=retry_state)
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 314, in iter
2024-08-06 04:13:29 [INFO] return fut.result()
2024-08-06 04:13:29 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
2024-08-06 04:13:29 [INFO] return self.__get_result()
2024-08-06 04:13:29 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2024-08-06 04:13:29 [INFO] raise self._exception
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __call__
2024-08-06 04:13:29 [INFO] result = fn(*args, **kwargs)
2024-08-06 04:13:29 [INFO] File "/mnt/git/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/octavia/tests.py", line 305, in wait_for_lb_resource
2024-08-06 04:13:29 [INFO] raise LoadBalancerUnrecoverableError(msg)
2024-08-06 04:13:29 [INFO] zaza.openstack.utilities.exceptions.LoadBalancerUnrecoverableError: load balancer resource has not reached expected provisioning status: {'id': '4e9c5798-ef75-4616-9ce6-b387f80c1f21', 'name': 'zaza-amphora-0', 'description': 'Created by Zaza', 'provisioning_status': 'ERROR', 'operating_status': 'OFFLINE', 'admin_state_up': True, 'project_id': '5ca71241795540c79860d5bf790b483a', 'created_at': '2024-08-06T04:13:15', 'updated_at': '2024-08-06T04:13:23', 'vip_address': '10.42.0.117', 'vip_port_id': '45322c47-c10c-4729-b9c3-888469a37b20', 'vip_subnet_id': '14e2ad19-2067-49c3-8da3-69904b9c36d7', 'vip_network_id': '602eb569-a566-4b32-b15b-9db93a22225b', 'additional_vips'...

[Felipe Reyes (freyes)]

Noble (Caracal) has been verified using Sunbeam, this is content of the octavia-api rock:

```
# ls /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/templates
amphora_agent_conf.template
```

The image correctly contains the template file (data file).

I built the rock using this patch - https://github.com/freyes/ubuntu-openstack-rocks/commit/fedda7dcb45a230e1a271e7993e3af75ad56de73

[Andreas Hasenack (ahasenack)]

Hi Felipe, thanks for the noble verification. Do you also have the results of what was asked in https://bugs.launchpad.net/ubuntu/+source/octavia/+bug/2072526/comments/5 ?