When internal tls and etcd are enabled, cinder fails to connect.

[Debasis]

When internal tls (self-signed) and etcd are enabled, cinder fails to connect with latest kolla ansible. I see the following error in the cinder volume and cinder-backup logs

Starting cinder-volume node (version 24.0.0)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service [-] Error starting thread.: etcd3gw.exceptions.Etcd3Exception: Not Found
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service Traceback (most recent call last):
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/service.py", line 810, in run_service
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service service.start()
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/service.py", line 227, in start
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service coordination.COORDINATOR.start()
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/coordination.py", line 87, in start
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service self.coordinator.start(start_heart=True)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/coordination.py", line 689, in start
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service super(CoordinationDriverWithExecutor, self).start(start_heart)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/coordination.py", line 426, in start
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service self._start()
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/drivers/etcd3gw.py", line 238, in _start
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service self._membership_lease = self.client.lease(self.membership_timeout)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 170, in lease
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service result = self.post(self.get_url("/lease/grant"),
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 145, in post
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service return self._request('post', *args, **kwargs)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 131, in _request
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service raise exceptions.Etcd3Exception(resp.text, resp.reason)
2024-06-03 19:55:19.871 11465 ERROR oslo_service.service etcd3gw.exceptions.Etcd3Exception: Not Found

[Przemysław Kuczyński (przemekkuczynski)]

Looks same as this https://bugs.launchpad.net/python-tooz/+bug/1983668

check comments under https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/cinder/templates/cinder.conf.j2

[Debasis (debamondal)]

After further investigation, what I observe is that if I don't use SSL and
point coordination backend_url (as given in cinder.conf) to the IP of the
node in /etc/hosts, there's no etcd3gw exception. But if I use the virtual
internal IP of the OpenStack created by the keepalive, I see that error.
So, I point the backend_url to the IP of the node to get rid off of the
exception. Once done, I apply SSL and now I see https error in the cinder
log. This goes away, once I point the backen_url back to the virtual IP of
the OpenStack given by the keepalive! However, this brings the etcd3gw
exception back! Those URLs you gave are of no use, I guess.

*Thanks and Regards,*
Debasis Mondal

On Tue, Jun 4, 2024 at 7:30 AM Przemysław Kuczyński <
<email address hidden>> wrote:

> Looks same as this https://bugs.launchpad.net/python-tooz/+bug/1983668
>
> check comments under https://github.com/openstack/kolla-
> ansible/blob/master/ansible/roles/cinder/templates/cinder.conf.j2
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2067914
>
> Title:
> When internal tls and etcd are enabled, cinder fails to connect.
>
> Status in kolla-ansible:
> New
>
> Bug description:
> When internal tls (self-signed) and etcd are enabled, cinder fails to
> connect with latest kolla ansible. I see the following error in the
> cinder volume and cinder-backup logs
>
> Starting cinder-volume node (version 24.0.0)
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service [-] Error
> starting thread.: etcd3gw.exceptions.Etcd3Exception: Not Found
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service Traceback (most
> recent call last):
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/service.py",
> line 810, in run_service
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service
> service.start()
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/service.py", line
> 227, in start
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service
> coordination.COORDINATOR.start()
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/coordination.py",
> line 87, in start
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service
> self.coordinator.start(start_heart=True)
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/coordination.py",
> line 689, in start
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service
> super(CoordinationDriverWithExecutor, self).start(start_heart)
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/coordination.py",
> line 426, in start
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service
> self._start()
> 2024-06-03 19:55:19.871 11465 ERROR oslo_service.service File
> "/var/lib/kolla/venv/lib/python3.10/site-packages/tooz/drivers/etcd3gw.py",
> line 238, ...