Mistral not starting on devstack

[Arnaud Morin]

When running devstack, following the documentation:
https://docs.openstack.org/mistral/latest/contributor/devstack.html

It fails with a stacktrace in systemd:

Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 ERROR oslo_service.service [-] Error starting thread.: etcd3gw.exceptions.Etcd3Exception: Not Found
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service Traceback (most recent call last):
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service service.start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/engine/engine_server.py", line 60, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service super(EngineServer, self).start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/base.py", line 46, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self.cluster_member.register_membership()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service return f(*args, **kwargs)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 174, in register_membership
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service service_coordinator = get_service_coordinator()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 156, in get_service_coordinator
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service _SERVICE_COORDINATOR.start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 54, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._coordinator.start(start_heart=True)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/coordination.py", line 689, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service super(CoordinationDriverWithExecutor, self).start(start_heart)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/coordination.py", line 426, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/drivers/etcd3gw.py", line 235, in _start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._membership_lease = self.client.lease(self.membership_timeout)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 122, in lease
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service result = self.post(self.get_url("/lease/grant"),
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 92, in post
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service raise exceptions.Etcd3Exception(resp.text, resp.reason)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service etcd3gw.exceptions.Etcd3Exception: Not Found
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service

This seems to be the root cause of the gate failing recently

[Arnaud Morin (arnaud-morin)]

I found the root cause, etcd3 endpoint is not configured correctly in mistral config on devstack.

This is the same issue as this one for cinder:

https://review.opendev.org/c/openstack/devstack/+/891353

[Jan Gutter (jangutter)]

Apologies, yeah, etcdv3 has dropped support for the deprecated endpoint and tooz couldn't be updated in time to make the default transparent.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/mistral/+/899169

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/mistral/+/899169
Committed: https://opendev.org/openstack/mistral/commit/e343ccb078d8ba261ac70afca93f4358589730d3
Submitter: "Zuul (22348)"
Branch: master

commit e343ccb078d8ba261ac70afca93f4358589730d3
Author: Arnaud Morin <email address hidden>
Date: Tue Oct 24 15:14:53 2023 +0200

    Add api_version for etcd3 coordination URL

    Since upgrade of etcd3 (see [1]), the v3aplha endpoint has been
    deprecated but tooz commit (see [2]) is not yet released.
    Until then, we should force the api_version=v3 to make sure the
    coordination between engine is working correctly.

    Closes-Bug: #2040048

    Signed-off-by: Arnaud Morin <email address hidden>
    Change-Id: Ieade7a30a192910fc3870b173017efd481f0b18f

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 18.0.0.0rc1 release candidate.