vcenter-as-compute: 16-04/newton : SM provisioning fails @ TASK : add esxi-host/dvs

[Sarath]

Version: 4.0.1.0-40-newton
Topology: 3node HA with multiple computes (multi-cluster Esxi) & Kvm

Kamlesh debugged the problem setup and knows the issue and already patched private fix on the setup to proceed further..

This issue seen when provisioning adding esxi and dvs to Vsphere.

"2017-09-01 00:03:17,048-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Wait for add host task complete [{u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/587006564114.29541', u'ansible_job_id': u'587006564114.29541', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/976186852647.29577', u'ansible_job_id': u'976186852647.29577', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/377271313251.29610', u'ansible_job_id': u'377271313251.29610', 'changed': True}]]"
"2017-09-01 00:03:17,106-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:19,473-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:21,827-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:24,175-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:26,528-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:28,879-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:31,229-DEBUG-server_mgr_main.py:1270-validate_smgr_request(): validate_smgr_request"
"2017-09-01 00:03:32,815-INFO-sm_ansible_callback.py:43-append(): changed: [localhost] => (item={u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/587006564114.29541', u'ansible_job_id': u'587006564114.29541', 'changed': True})"
"2017-09-01 00:03:32,987-INFO-sm_ansible_callback.py:43-append(): changed: [localhost] => (item={u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/976186852647.29577', u'ansible_job_id': u'976186852647.29577', 'changed': True})"
"2017-09-01 00:03:33,126-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]: FAILED! => (item - None) {"changed": true, "failed": true, "msg": "One or more items failed"}"
"2017-09-01 00:03:33,130-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost] (item={u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/377271313251.29610', u'ansible_job_id': u'377271313251.29610', 'changed': True}) => {"ansible_job_id": "377271313251.29610", "attempts": 1, "changed": false, "failed": true, "finished": 1, "item": {"ansible_job_id": "377271313251.29610", "c

Vsphere also has exception registered part of this ansible task as below,
"Cannot complete operation due to concurrent modifcation by another operation"

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35221
Submitter: kamlesh parmar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35222
Submitter: kamlesh parmar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35228
Submitter: kamlesh parmar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35222
Submitter: kamlesh parmar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35222
Committed: http://github.com/Juniper/contrail-server-manager/commit/de4c0f93a05e24e25675b5bd3d33939ddcdaeb05
Submitter: Zuul (<email address hidden>)
Branch: master

commit de4c0f93a05e24e25675b5bd3d33939ddcdaeb05
Author: Kamlesh Parmar <email address hidden>
Date: Sat Sep 2 14:19:24 2017 -0700

Substitute the vcenter/dc/dvs which has cluster same as the esxi host.
update the schema file for missing key.

Change-Id: I6c67e1dd1378edbbac272cdbff875ad001004e72
Closes-Bug: #1714713

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35228
Committed: http://github.com/Juniper/contrail-ansible/commit/5f9192e8dd845f4eef4e853d13a276244abf3d2c
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 5f9192e8dd845f4eef4e853d13a276244abf3d2c
Author: Kamlesh Parmar <email address hidden>
Date: Sun Sep 3 14:41:33 2017 -0700

Adding multiple hosts to dvs concurrently is not supported in vcenter.
remove parallel tasks for add host to dvs.

Change-Id: Id872f829094ea595d8bae153aad935f6f2041bfa
Closes-Bug: #1714713

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35221
Committed: http://github.com/Juniper/contrail-server-manager/commit/75db1da01b8c7b76fbefbfe04b7ebb64b60eacb7
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 75db1da01b8c7b76fbefbfe04b7ebb64b60eacb7
Author: Kamlesh Parmar <email address hidden>
Date: Sat Sep 2 14:19:24 2017 -0700

Substitute the vcenter/dc/dvs which has cluster same as the esxi host.
update the schema file for missing key.

Change-Id: I6c67e1dd1378edbbac272cdbff875ad001004e72
Closes-Bug: #1714713