kolla-ansible

Coordination not working when redis used

Bug #2056667 reported by Michal Arbet on 2024-03-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Fix Released	Undecided	Michal Arbet

Bug Description

Hi,

New tooz 6.0.1 broke coordination via redis, log from cinder :

2024-03-10 00:50:15.890 23382 ERROR oslo_service.service Traceback (most recent call last):
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/drivers/redis.py", line 53, in wrapper
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service return func(*args, **kwargs)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/drivers/redis.py", line 501, in _start
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service self._server_info = self._client.info()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/commands/core.py", line 1002, in info
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service return self.execute_command("INFO", **kwargs)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service conn = self.connection or pool.get_connection(command_name, **options)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/connection.py", line 1086, in get_connection
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service connection.connect()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/sentinel.py", line 55, in connect
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service return self.retry.call_with_retry(self._connect_retry, lambda error: None)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/retry.py", line 51, in call_with_retry
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service raise error
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service return do()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/sentinel.py", line 45, in _connect_retry
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service self.connect_to(self.connection_pool.get_master_address())
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/sentinel.py", line 107, in get_master_address
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service master_address = self.sentinel_manager.discover_master(self.service_name)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/redis/sentinel.py", line 301, in discover_master
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service raise MasterNotFoundError(f"No master found for {service_name!r}{error_info}")
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service redis.sentinel.MasterNotFoundError: No master found for 'kolla' : Redis<ConnectionPool<Connection<host=192.168.205.10,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.'), Redis<ConnectionPool<Connection<host=192.168.205.11,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.'), Redis<ConnectionPool<Connection<host=192.168.205.12,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.')
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service The above exception was the direct cause of the following exception:
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service Traceback (most recent call last):
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/service.py", line 810, in run_service
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service service.start()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/cinder/service.py", line 227, in start
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service coordination.COORDINATOR.start()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/cinder/coordination.py", line 87, in start
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service self.coordinator.start(start_heart=True)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/coordination.py", line 689, in start
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service super(CoordinationDriverWithExecutor, self).start(start_heart)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/coordination.py", line 426, in start
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service self._start()
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/drivers/redis.py", line 62, in wrapper
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service utils.raise_with_cause(
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/tooz/utils.py", line 223, in raise_with_cause
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_utils/excutils.py", line 142, in raise_with_cause
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service raise exc_cls(message, *args, **kwargs) from kwargs.get('cause')
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service tooz.coordination.ToozConnectionError: No master found for 'kolla' : Redis<ConnectionPool<Connection<host=192.168.205.10,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.'), Redis<ConnectionPool<Connection<host=192.168.205.11,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.'), Redis<ConnectionPool<Connection<host=192.168.205.12,port=26379,db=0>>> - AuthenticationError('invalid username-password pair or user is disabled.')
2024-03-10 00:50:15.890 23382 ERROR oslo_service.service
2024-03-10 00:50:15.914 6 INFO oslo_service.service [None req-6c60300f-11a4-4823-9f45-454135e22423 - - - - - -] Child 23382 exited with status 1
2024-03-10 00:50:15.939 23383 INFO cinder.service [-] Starting cinder-volume node (version 23.1.0)

Michal Arbet (michalarbet) on 2024-03-10

Changed in kolla-ansible:
assignee:	nobody → Michal Arbet (michalarbet)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-10: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912341

Changed in kolla-ansible:
status:	New → In Progress

Revision history for this message

jiveg94665 (jiveg94665) wrote on 2024-03-10:

"""Kodexo Labs offers top-tier offshore software development services, catering to diverse business needs globally. With a focus on innovation and expertise, our team delivers scalable and efficient solutions tailored to your requirements, ensuring quality outcomes and cost-effective results. Partner with us to leverage cutting-edge technology and streamline your software development process.""" <a href="https://kodexolabs.com/offshore-software-development-company/">offshore software development</a>

Revision history for this message

jiveg94665 (jiveg94665) wrote on 2024-03-10:

[offshore software development](https://kodexolabs.com/offshore-software-development-company/) """Kodexo Labs offers top-tier offshore software development services, catering to diverse business needs globally. With a focus on innovation and expertise, our team delivers scalable and efficient solutions tailored to your requirements, ensuring quality outcomes and cost-effective results. Partner with us to leverage cutting-edge technology and streamline your software development process."""

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-11: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912341
Committed: https://opendev.org/openstack/kolla-ansible/commit/59da07920be5eb561446c0ead08bf870263f0cd5
Submitter: "Zuul (22348)"
Branch: master

commit 59da07920be5eb561446c0ead08bf870263f0cd5
Author: Michal Arbet <email address hidden>
Date: Sun Mar 10 10:39:31 2024 +0100

Fix coordination when redis used

    Tooz 6.0.1 includes commit [1], which introduced
    parsing the username from the Redis connection URL.
    As a result, services started authenticating as admin
    which, by the way, was incorrect even before, as either
    a created user or the default one should have been used.

The reason it worked before is simply because the username
'admin' wasn't parsed anywhere.

This patch fixes the user being used and sets the correct
'default' one.

[1] https://review.opendev.org/c/openstack/tooz/+/907656

    Closes-Bug: #2056667
    Depends-On: https://review.opendev.org/c/openstack/kolla/+/911703
    Change-Id: I5568dba15fa98e009ad4a9e41756aba0fa659371

Changed in kolla-ansible:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.