kolla-ansible

Octavia is failing because of missing config

Bug #2073775 reported by Michal Arbet
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
In Progress
Undecided
Michal Arbet

Bug Description

Same problem as for heat https://bugs.launchpad.net/kolla-ansible/+bug/2073760

/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils [-] Unhandled exception: oslo_config.cfg.RequiredOptError: <exception str() failed>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils Traceback (most recent call last):
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils yield
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/cotyledon/_service.py", line 139, in _run
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils self.run()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/octavia/controller/queue/v2/consumer.py", line 52, in run
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils self.message_listener.start()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 267, in wrapper
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 188, in run_once
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils post_fn = fn()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 267, in <lambda>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 413, in start
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils self.listener = self._create_listener()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils return self.transport._listen(self._target, 1, None)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/transport.py", line 142, in _listen
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils return self._driver.listen(target, batch_size,
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 802, in listen
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils conn.declare_fanout_consumer(target.topic, listener)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1477, in declare_fanout_consumer
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils unique = self._q_manager.get()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 98, in get
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils c = read_from_shm()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 403, in inner
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils with lock(name, lock_file_prefix, external, lock_path,
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils return next(self.gen)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 316, in lock
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ext_lock = external_lock(name, lock_file_prefix, lock_path)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 206, in external_lock
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils lock_file_path = _get_lock_path(name, lock_file_prefix, lock_path)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 200, in _get_lock_path
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils raise cfg.RequiredOptError('lock_path', 'oslo_concurrency')
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils oslo_config.cfg.RequiredOptError: <exception str() failed>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.286 1692 ERROR cotyledon._utils
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils [-] Unhandled exception: oslo_config.cfg.RequiredOptError: <exception str() failed>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils Traceback (most recent call last):
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils yield
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/cotyledon/_service.py", line 139, in _run
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils self.run()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/octavia/controller/queue/v2/consumer.py", line 52, in run
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils self.message_listener.start()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 267, in wrapper
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 188, in run_once
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils post_fn = fn()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 267, in <lambda>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/server.py", line 413, in start
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils self.listener = self._create_listener()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils return self.transport._listen(self._target, 1, None)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/transport.py", line 142, in _listen
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils return self._driver.listen(target, batch_size,
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 802, in listen
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils conn.declare_fanout_consumer(target.topic, listener)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1477, in declare_fanout_consumer
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils unique = self._q_manager.get()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 98, in get
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils c = read_from_shm()
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 403, in inner
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils with lock(name, lock_file_prefix, external, lock_path,
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils return next(self.gen)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 316, in lock
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ext_lock = external_lock(name, lock_file_prefix, lock_path)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 206, in external_lock
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils lock_file_path = _get_lock_path(name, lock_file_prefix, lock_path)
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_concurrency/lockutils.py", line 200, in _get_lock_path
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils raise cfg.RequiredOptError('lock_path', 'oslo_concurrency')
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils oslo_config.cfg.RequiredOptError: <exception str() failed>
/var/log/kolla/octavia/octavia-worker.log:2024-07-22 15:02:18.400 1695 ERROR cotyledon._utils

Revision history for this message
Michal Arbet (michalarbet) wrote :

tested on 2024.1

Changed in kolla-ansible:
assignee: nobody → Michal Arbet (michalarbet)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/924652

Changed in kolla-ansible:
status: New → In Progress
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

How is your setup configured compared to the CI because we don't seem to see the issue there (or do we and you just didn't mention it?)

Revision history for this message
Michal Arbet (michalarbet) wrote :

I think it's related to oslo.messaging and use of queue manager which is really nice feature , here you can see it's problem

https://458b7eabeaed9e7bde26-c6cd00e15301f9cbec535c910ea1d6bc.ssl.cf2.rackcdn.com/924623/1/check/kolla-ansible-debian/93d615d/primary/logs/kolla/heat/heat-engine.txt

Review - https://review.opendev.org/c/openstack/kolla-ansible/+/924623

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Ah, yes, the code that triggers this has this note:

https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/amqpdriver.py#L47-L48

But to me this implies, that this is not an issue in k-a currently, but just something that needs to be done as part of https://review.opendev.org/c/openstack/kolla-ansible/+/924623

Revision history for this message
Michal Arbet (michalarbet) wrote :

Yeah, actually it's not a bug currently as kolla don't use queue manager. But it is super nice feature of oslo.messaging. I can squash those commits into one, modity the reno to add features block. Should I somehow close this bugreports or add just related-bug ? Or how ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by "Michal Arbet <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/924652

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.