Version: 4.0.1.0-69-mitaka
Topology: 3node HA with multiple computes (multi-cluster Esxi) & Kvm
This is regression bug which used to work in #50 build last tested.
this looks broken in Mitaka possibly due to below bug commit for Newton,
---------
add unverified ssl context for urllib request. This is required for
xenial. Daemonize the keepalive thread.
Change-Id: Id2a8d8447209a29e6f35c3d2dcc44bee84bb795f
Closes-Bug: #1714854
------------
Please find below the debugging / Tracebacks,
"2017-09-10 01:49:26,812-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Create contrail vm in the esxi host {'username': u'root', 'datacenter': u'A6s32-VC-Compute-Sanity', 'std_switch_list': [{'pg_name': u'mgmt-pg', 'switch_name': u'vSwitch0'}], 'name': u'10.87.36.29', 'contrail_vm': {'vmdk': u'/root/vmdk/vmdk.tar', 'mode': u'vcenter', 'mgmt_pg': u'mgmt-pg', 'networks': [{'mac': u'00:77:56:aa:bb:03', 'sw_type': u'standard', 'role': u'mgmt', 'switch_name': u'vSwitch0', 'pg': u'mgmt-pg'}]}, 'cluster': u'cluster-2', 'id_in_sm': u'contrailvm-5a10s31', 'datastore': u'datastore4', 'password': u'c0ntrail123', 'validate_certs': False, 'skip_reimage': u'true', 'vcenter_server': {'username': <email address hidden>', 'datacentername': u'A6s32-VC-Compute-Sanity', 'password': u'Contrail123!', 'vcenter_compute_ip': u'10.87.36.27', 'hostname': u'10.87.131.15', 'dv_port_group_mgmt': None, 'dv_switch_control_data': {'dv_switch_name': None}, 'dv_switch_mgmt': {'dv_switch_name': None}, 'clusternames': [u'cluster-2'], 'dv_switch': {'dv_switch_name': u'Sanity-dvswitch-2'}, 'dv_port_group_control_data': None, 'validate_certs': False, 'dv_port_group': {'dv_portgroup_name': u'Sanity-dvportgroup-2', 'number_of_ports': u'1800'}}}]"
"2017-09-10 01:49:27,938-INFO-sm_ansible_callback.py:43-append(): changed: [localhost]"
"2017-09-10 01:49:27,950-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Save task results {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/247251544613.23958', u'ansible_job_id': u'247251544613.23958', 'changed': True}]"
"2017-09-10 01:49:27,989-INFO-sm_ansible_callback.py:43-append(): ok: [localhost]"
"2017-09-10 01:49:28,004-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Wait for contrailVm create completion [{u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/515287490246.23884', u'ansible_job_id': u'515287490246.23884', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/779700746384.23927', u'ansible_job_id': u'779700746384.23927', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/247251544613.23958', u'ansible_job_id': u'247251544613.23958', 'changed': True}]]"
"2017-09-10 01:49:32,739-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:49:32,739-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:49:33,369-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
exit(main())
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,523-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
exit(main())
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,686-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
exit(main())
File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,689-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]: FAILED! => (item - None) {"changed": true, "failed": true, "msg": "One or more items failed"}"
"2017-09-10 01:49:33,694-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=localhost&state=provision_failed"
"2017-09-10 01:49:33,697-DEBUG-server_mgr_status.py:111-put_ansible_status(): Server status Data provision_failed localhost 2017_09_10__01_49_33"
"2017-09-10 01:49:33,699-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [localhost] : ok: 70 changed: 18 unreachable: 0 skipped: 17 failed: 1"
"2017-09-10 01:49:33,702-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.10&state=provision_failed"
"2017-09-10 01:49:33,704-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s31 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:33,828-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:33,831-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.11&state=provision_failed"
"2017-09-10 01:49:33,834-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s30 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:33,962-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:33,964-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:49:33,965-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:49:33,965-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.12&state=provision_failed"
"2017-09-10 01:49:33,967-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s29 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:34,096-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,099-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.18&state=provision_failed"
"2017-09-10 01:49:34,101-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s23 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,221-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,224-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.19&state=provision_failed"
"2017-09-10 01:49:34,227-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s27 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,347-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,350-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.20&state=provision_failed"
"2017-09-10 01:49:34,353-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s25 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,687-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,690-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.21&state=provision_failed"
"2017-09-10 01:49:34,692-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s31 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,807-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,810-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.27&state=provision_failed"
"2017-09-10 01:49:34,812-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s01b provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,924-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,927-ERROR-sm_ansible_playbook.py:778-run_playbook(): Playbook Failed: /opt/contrail/server_manager/ansible/playbooks/contrail_4_0_1_50_vc_new/playbooks/site.yml"
"2017-09-10 01:49:34,937-INFO-sm_ansible_server.py:42-run(): Process Done"
"2017-09-10 01:49:38,691-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:49:38,691-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:49:39,669-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.11, option => key"
"2017-09-10 01:50:17,688-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.10, option => key"
"2017-09-10 01:50:35,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:50:35,695-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:50:36,315-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:50:36,315-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:50:36,879-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:50:36,879-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:50:58,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => key, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:35,739-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:35,740-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:36,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => key, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:51:36,971-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:51:36,972-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:51:39,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:51:39,511-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:52:11,511-DEBUG-server_mgr_mon_base_plugin.py:706-create_server_dict(): Created server dictionary."
"2017-09-10 01:52:11,665-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.18, option => key"
"2017-09-10 01:52:38,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:52:38,695-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 29: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
Review in progress for https:/ /review. opencontrail. org/35455
Submitter: kamlesh parmar (<email address hidden>)