[18.04][Rocky->Stein] Upgrade to Stein resulted in service "designate-producer" can't start

[Jan Wasilewski]

I was progressing with upgrade of designate from Rocky to Stein release. Everything seems to work fine except designate-producer which can't start after upgrade. There is an error code like below during startup:

2021-05-12 16:45:15.918 8268 INFO designate.metrics [-] Statsd disabled
2021-05-12 16:45:15.992 8268 INFO designate.service [-] Starting producer service (version: 8.0.1)
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service [-] Error starting thread.: ValueError: 'designate-01:a239d340-a094-4312-9ae8-cf53ceb6343c' is not in list
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service Traceback (most recent call last):
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service service.start()
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/producer/service.py", line 72, in start
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service self._partitioner.start()
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 223, in start
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service self._my_partitions = self._update_partitions()[1]
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 202, in _update_partitions
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service members, self._my_id, self._partitions)
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/designate/coordination.py", line 187, in _partition
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service my_index = members.index(me)
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service ValueError: 'designate-01:a239d340-a094-4312-9ae8-cf53ceb6343c' is not in list
2021-05-12 16:45:16.080 8268 ERROR oslo_service.service
2021-05-12 16:45:16.090 8268 INFO designate.service [-] Stopping producer service

I was trying to implement similar solution like from bug: https://bugs.launchpad.net/charm-designate/+bug/1828534 but it didn't help.

[System details]
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Packages before upgrade:
ii designate 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - metapackage
ii designate-agent 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - agent
ii designate-api 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - API server
ii designate-central 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - central daemon
ii designate-common 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - common files
ii designate-mdns 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - mdns
ii designate-producer 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - producer
ii designate-sink 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - sink
ii designate-worker 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - worker
ii python-designate 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - Python 2 libs
ii python-designateclient 2.9.0-0ubuntu1 all client library for the OpenStack Designate API - Python 2.7

Packages after upgrade:
ii designate 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - metapackage
ii designate-agent 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - agent
ii designate-api 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - API server
ii designate-central 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - central daemon
ii designate-common 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - common files
ii designate-mdns 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - mdns
ii designate-producer 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - producer
ii designate-sink 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - sink
ii designate-worker 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - worker
rc python-designate 1:7.0.0-0ubuntu1~cloud0 all OpenStack DNS as a Service - Python 2 libs
ii python-designateclient 2.9.0-0ubuntu1 all client library for the OpenStack Designate API - Python 2.7
ii python3-designate 1:8.0.1-0ubuntu1~cloud0 all OpenStack DNS as a Service - Python 3 libs
ii python3-designateclient 2.9.0-0ubuntu1 all client library for the OpenStack Designate API - Python 3.x

Backend: bind9

It seems it can be related somehow with python version(2 vs. 3) but I couldn't find more details.
If some additional logs are needed - please let me know, I will collect this.

[Patryk Jakuszew (pjakuszew)]

We have located the root cause of this issue. It is caused by the following bug:

https://bugs.launchpad.net/designate/+bug/1828534

I don't quite understand why it wasn't backported back to Stein packages...

[Patryk Jakuszew (pjakuszew)]

To be more specific, this commit fixes the issue on our Python3-based Stein deployment: https://opendev.org/openstack/designate/commit/4ad8defd4a4ad0f5b0d29de34a3b6e9e90aa4b43