Leader is determined by lowest sorted unit name, but sorted alphanumerically

[Ryan Finnie]

An environment has three u-r-c units, ubuntu-repository-cache/{9,10,11}. ubuntu-repository-cache/9 takes care of the master syncs, but the code for determining which relation to get meta_version from sorts alphanumerically and checks ubuntu-repository-cache/10, resulting in "Metadata version not yet provided by leader" on 10 and 11. 9 ends up going live anyway, resulting in a mismatch between 9 and 10/11.

ubuntu-repository-cache/9:

2016-08-15 18:54:51 DEBUG juju-log Pushing canary files to peers
2016-08-15 18:54:51 INFO juju-log Syncing /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_18:51:01_u9_good to www-sync@172.31.28.198:/srv/ubuntu-repository-cache/ephemeral/0/apache/data.
2016-08-15 18:54:51 INFO juju-log /usr/bin/rsync -r --quiet --recursive --links --perms --chmod=g+w --times --compress --delete --delete-during --timeout=120 -e sudo -u www-sync ssh -l www-sync /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_18:51:01_u9_good www-sync@172.31.28.198:/srv/ubuntu-repository-cache/ephemeral/0/apache/data
2016-08-15 18:54:52 INFO juju-log Sync to 172.31.28.198 complete.
2016-08-15 18:54:52 INFO juju-log Syncing /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_18:51:01_u9_good to www-sync@172.31.30.254:/srv/ubuntu-repository-cache/ephemeral/0/apache/data.
2016-08-15 18:54:52 INFO juju-log /usr/bin/rsync -r --quiet --recursive --links --perms --chmod=g+w --times --compress --delete --delete-during --timeout=120 -e sudo -u www-sync ssh -l www-sync /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_18:51:01_u9_good www-sync@172.31.30.254:/srv/ubuntu-repository-cache/ephemeral/0/apache/data
2016-08-15 18:54:53 INFO juju-log Sync to 172.31.30.254 complete.
2016-08-15 18:54:53 DEBUG juju-log
2016-08-15 18:54:53 DEBUG juju-log Updating meta_version for ids ['cluster:3']
2016-08-15 18:54:53 DEBUG juju-log Setting meta_version=ubuntu_2016-08-15_18:51:01_u9
2016-08-15 18:54:53 INFO juju-log Symlinking /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_18:51:01_u9 as /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_next
2016-08-15 18:54:53 INFO juju-log Cleaning metadata directory
2016-08-15 18:54:53 DEBUG juju-log Removing tree /srv/ubuntu-repository-cache/ephemeral/0/apache/data/ubuntu_2016-08-15_17:51:01_u9
2016-08-15 18:54:54 INFO juju-log Starting services
2016-08-15 18:54:54 INFO juju-log Starting squid
2016-08-15 18:54:54 INFO juju-log Starting apache

ubuntu-repository-cache/10:

2016-08-15 18:54:58 INFO juju-log cluster:3: Cluster relation changed for ubuntu-repository-cache
2016-08-15 18:54:58 INFO juju-log cluster:3: SSH key already exists at /home/www-sync/.ssh/id_rsa.
2016-08-15 18:54:58 INFO juju-log cluster:3: Syncing authorized_keys @ /home/www-sync/.ssh/authorized_keys.
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.30.254:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.30.254:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.30.254:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.0.120:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.0.120:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:58 INFO cluster-relation-changed # 172.31.0.120:22 SSH-2.0-OpenSSH_7.1p2 Ubuntu-2.IS.PATCHED.16.04.4
2016-08-15 18:54:59 INFO juju-log cluster:3: Syncing known_hosts @ /home/www-sync/.ssh/known_hosts.
2016-08-15 18:54:59 INFO juju-log cluster:3: Deferring action to oldest service unit.
2016-08-15 18:54:59 INFO juju-log cluster:3: Updating metadata on a peer
2016-08-15 18:54:59 INFO juju-log cluster:3: Metadata version not yet provided by leader.
2016-08-15 18:54:59 INFO juju-log cluster:3: Rendering configuration templates
2016-08-15 18:54:59 INFO juju-log cluster:3: Rendering apache2 configuration templates
2016-08-15 18:54:59 INFO juju-log cluster:3: Deferring action to oldest service unit.
2016-08-15 18:54:59 INFO juju-log cluster:3: Writing file /etc/apache2/sites-available/archive_ubuntu_com.conf root:root 444
2016-08-15 18:54:59 INFO juju-log cluster:3: Writing file /etc/apache2/conf-available/000mpm-worker.conf root:root 444
2016-08-15 18:54:59 INFO cluster-relation-changed Conf 000mpm-worker already enabled
2016-08-15 18:54:59 INFO juju-log cluster:3: Writing file /etc/apache2/conf-available/security.conf root:root 444
2016-08-15 18:54:59 INFO cluster-relation-changed Conf security already enabled
2016-08-15 18:54:59 INFO juju-log cluster:3: Rendering squid configuration templates
2016-08-15 18:55:00 INFO juju-log cluster:3: Writing file /etc/squid-deb-proxy/squid-deb-proxy.conf root:root 444
2016-08-15 18:55:00 INFO juju-log cluster:3: Writing file /etc/squid-deb-proxy/allowed-networks-src.acl root:root 444
2016-08-15 18:55:00 INFO juju-log cluster:3: Writing file /etc/logrotate.d/apache2 root:root 444
2016-08-15 18:55:00 INFO juju-log cluster:3: Deferring action to oldest service unit.

[Paul Gear (paulgear)]

Note that the solution to this should probably be used to fix #1559230 and #1576920 in the process.

[Chris Glass (tribaal)]

This should not happen anymore as we switched to using juju-provided leader-election instead.