Reproducing with designate, the problem appears after we have one designate unit that is at bionic-rocky (py3) and one that is at bionic-queens (py2). It is resolved once all units are at bionic-rocky. More details on that:
# upgrade one unit to bionic-rocky
juju config designate openstack-origin=cloud:bionic-rocky
juju run-action designate/0 openstack-upgrade
juju status
Unit Workload Agent Machine Public address Ports Message
designate/0 error idle 7 10.5.0.198 9001/tcp Services not running that should be: designate-producer
hacluster-designate/0 active idle 10.5.0.198 Unit is ready and clustered
designate/1* active idle 8 10.5.0.6 9001/tcp Unit is ready
hacluster-designate/1* active idle 10.5.0.6 Unit is ready and clustered
where:
- machine 8 (b'juju-3d28eb-coreycb2-8:86c49114-a37a-4e1a-8654-72bf1ccbde1f') hasn't been upgraded yet and is still bionic-queens
- machine 7 ('juju-3d28eb-coreycb2-7:d64cde68-d9f0-4910-9f1a-6d9724ea77b9') is the machine that was upgraded to bionic-rocky
Note: designate coordinator backend gets the juju strings by calling get_members() from the tooz memcached backend. /usr/lib/python3/dist-packages/tooz/drivers/memcached.py
# now try to upgrade the other unit to bionic-rocky
juju run-action designate/1 openstack-upgrade
# restart memcache and designate-producer on designate/0 and resolve
juju resolved designate/0
# all better
Unit Workload Agent Machine Public address Ports Message
designate/0 active idle 7 10.5.0.198 9001/tcp Unit is ready
hacluster-designate/0 active idle 10.5.0.198 Unit is ready and clustered
designate/1* active idle 8 10.5.0.6 9001/tcp Unit is ready
hacluster-designate/1* active idle 10.5.0.6 Unit is ready and clustered
Reproducing with designate, the problem appears after we have one designate unit that is at bionic-rocky (py3) and one that is at bionic-queens (py2). It is resolved once all units are at bionic-rocky. More details on that:
# upgrade one unit to bionic-rocky origin= cloud:bionic- rocky
juju config designate openstack-
juju run-action designate/0 openstack-upgrade
juju status designate/ 0 active idle 10.5.0.198 Unit is ready and clustered designate/ 1* active idle 10.5.0.6 Unit is ready and clustered
Unit Workload Agent Machine Public address Ports Message
designate/0 error idle 7 10.5.0.198 9001/tcp Services not running that should be: designate-producer
hacluster-
designate/1* active idle 8 10.5.0.6 9001/tcp Unit is ready
hacluster-
juju ssh designate/0
2019-07-25 18:04:27.452 24807 DEBUG designate. coordination [-] CCB: list(self. _get_members( self._group_ id))=[b' juju-3d28eb- coreycb2- 8:86c49114- a37a-4e1a- 8654-72bf1ccbde 1f', 'juju-3d28eb- coreycb2- 7:d64cde68- d9f0-4910- 9f1a-6d9724ea77 b9'] _update_partitions /usr/lib/ python3/ dist-packages/ designate/ coordination. py:202
where: 3d28eb- coreycb2- 8:86c49114- a37a-4e1a- 8654-72bf1ccbde 1f') hasn't been upgraded yet and is still bionic-queens 3d28eb- coreycb2- 7:d64cde68- d9f0-4910- 9f1a-6d9724ea77 b9') is the machine that was upgraded to bionic-rocky
- machine 8 (b'juju-
- machine 7 ('juju-
Note: designate coordinator backend gets the juju strings by calling get_members() from the tooz memcached backend. /usr/lib/ python3/ dist-packages/ tooz/drivers/ memcached. py
# now try to upgrade the other unit to bionic-rocky
juju run-action designate/1 openstack-upgrade
# restart memcache and designate-producer on designate/0 and resolve
juju resolved designate/0
# all better
Unit Workload Agent Machine Public address Ports Message designate/ 0 active idle 10.5.0.198 Unit is ready and clustered designate/ 1* active idle 10.5.0.6 Unit is ready and clustered
designate/0 active idle 7 10.5.0.198 9001/tcp Unit is ready
hacluster-
designate/1* active idle 8 10.5.0.6 9001/tcp Unit is ready
hacluster-
juju ssh designate/0
2019-07-25 19:31:11.365 12217 DEBUG designate. coordination [-] CCB: list(self. _get_members( self._group_ id))=[' juju-3d28eb- coreycb2- 7:0ceca5d8- 9eee-492d- bead-f99ee990a2 1b', 'juju-3d28eb- coreycb2- 8:134e6142- 4364-43a5- 96fc-b9f125ba87 a0'] _update_partitions /usr/lib/ python3/ dist-packages/ designate/ coordination. py:202
Both strings are str type now and able to be sorted.