HA provisioning with fab fails if config nodes in testbed.py are not is sequential order ( The chosen master node is not the first node)

[Vivek]

testbed.py with failed configuration
-------------------------------------------------------------------------------------------------
from fabric.api import env
import os
#Management ip addresses of hosts in the cluster
host1 = 'root@10.87.140.27'
host2 = 'root@10.87.140.48'
host3 = 'root@10.87.129.168'
host4 = 'root@10.87.129.59'
host5 = 'root@10.87.140.49'
#External routers if any
ext_routers = [('montreal', '10.87.140.140')]

#Autonomous system number
router_asn = 64512
public_vn_rtgt = 10000
public_vn_subnet = "10.87.129.224/27"

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.87.140.27'

#Role definition of the hosts.
env.roledefs = {
    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5],
    'collector': [host1,host2,host3],
    'webui': [host1,host2,host3],
    'database': [host1,host2,host3],
    'storage-master': [host1,host2,host3],
    'storage-compute': [ host4, host5],
    'build': [host_build],
}

env.hostnames = {
    'all': [ 'cmbu-vxa20-15','cmbu-vxa22-19', 'cmbu-vxa20-2','cmbu-vxa22-9','cmbu-vxa20-4' ]
}
--------------------------------------------------------------------------------------------

During fab setup_all, the IPs got sorted and master node was chosen to be host3 10.87.129.168 and the VIP were assigned to this node

This led to following error
2015-05-14 18:38:37:837959: [root@10.87.129.59] out: [localhost] local: chkconfig supervisor-vrouter on
2015-05-14 18:38:37:838152: [root@10.87.129.59] out: [localhost] local: python /opt/contrail/utils/provision_vrouter.py --host_name cmbu-vxa22-9 --host_ip 13.0.0.5 --api_server_ip 13.0.0.7 --oper add --admin_user admin --admin_password n1keenA --admin_tenant_name admin --openstack_ip 13.0.0.7
2015-05-14 18:38:37:838274: [root@10.87.129.59] out: Traceback (most recent call last):
2015-05-14 18:38:38:171540: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 186, in <module>
2015-05-14 18:38:38:171698: [root@10.87.129.59] out: main()
2015-05-14 18:38:38:171791: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 182, in main
2015-05-14 18:38:38:171881: [root@10.87.129.59] out: VrouterProvisioner(args_str)
2015-05-14 18:38:38:171968: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 32, in __init__
2015-05-14 18:38:38:172056: [root@10.87.129.59] out: auth_host=self._args.openstack_ip)
2015-05-14 18:38:38:172143: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 200, in __init__
2015-05-14 18:38:38:172264: [root@10.87.129.59] out: retry_on_error=False)
2015-05-14 18:38:38:172361: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 404, in _request
2015-05-14 18:38:38:172450: [root@10.87.129.59] out: content = self._request(op, url, data=data, retry_after_authn=True)
2015-05-14 18:38:38:172564: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 429, in _request
2015-05-14 18:38:38:172653: [root@10.87.129.59] out: raise HttpError(status, content)
2015-05-14 18:38:38:172741: [root@10.87.129.59] out: cfgm_common.exceptions.HttpError: HTTP Status: 401 Content: Authentication required
2015-05-14 18:38:38:172826: [root@10.87.129.59] out:
2015-05-14 18:38:38:189702: [root@10.87.129.59] out: Fatal error: local() encountered an error (return code 1) while executing 'python /opt/contrail/utils/provision_vrouter.py --host_name cmbu-vxa22-9 --host_ip 13.0.0.5 --api_server_ip 13.0.0.7 --oper add --admin_user admin --admin_password n1keenA --admin_tenant_name admin --openstack_ip 13.0.0.7'
2015-05-14 18:38:38:189806: [root@10.87.129.59] out:
2015-05-14 18:38:38:189889: [root@10.87.129.59] out: Aborting.
2015-05-14 18:38:38:189954: [root@10.87.129.59] out:

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/10618
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/10619
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/10618
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/53c69f193cc2e85918d2f42fc616705ece26d8e2
Submitter: Zuul
Branch: R2.20

commit 53c69f193cc2e85918d2f42fc616705ece26d8e2
Author: Ranjeet R <email address hidden>
Date: Wed May 20 13:56:15 2015 -0700

Fixes: HA provisioning with fab fails if config nodes in testbed.py
are not is sequential order ( The chosen master node is not the first node)

While comparing the openstack and cfgm node sets, we are sorting the
nodes which changes the order in which they are mentioned.

Changing the compare to use sets instead of using sort.

Change-Id: I312664cb67ad51244cdcc5ae0eecdb739644a8df
ClosesBug: 1455698

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/10619
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/3e1c9cd9ce29379d90dca655401cd5b4e4258283
Submitter: Zuul
Branch: master

commit 3e1c9cd9ce29379d90dca655401cd5b4e4258283
Author: Ranjeet R <email address hidden>
Date: Wed May 20 14:01:19 2015 -0700

Fixes: HA provisioning with fab fails if config nodes in testbed.py
are not is sequential order ( The chosen master node is not the first node)

While comparing the openstack and cfgm node sets, we are sorting the
nodes which changes the order in which they are mentioned.

Changing the compare to use sets instead of using sort.

Change-Id: Iaa1577202965eeec899c8226321e8bd946f81844
ClosesBug: 1455698