Juniper Openstack

Vcenter-as-compute: 16-04 : contrail-deploy when failed connecting vcenter-server not shown on SM debug.log

Series r4.1
Bug #1728751

Bug #1728751 reported by Sarath on 2017-10-31

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.1	Fix Committed	Medium	Ramprakash R	Juniper Openstack r4.1.0.0-fcs "r4.1"
Trunk	Fix Committed	Medium	Ramprakash R	Juniper Openstack r5.0.0

Bug Description

This actual issue is due to missing 16-04 code which fails connectivity to Vcenter-server.
But SM debug.log to show the details of sequence which is missing.

>>>>>>>>> debug.log after contrail-deploy shows provisionig failed without listing the sub-tasks failure

"2017-10-29 23:26:46,997-INFO-sm_ansible_playbook.py:227-run(): Executing Ansible Playbook Actions: ['contrail_deploy']"
"2017-10-29 23:26:47,000-INFO-sm_ansible_playbook.py:185-run_playbook(): Creating inventory /opt/contrail/server_manager/ansible/playbooks/ubuntu16newton30/playbooks/inventory/cluster-vcenter-compute.inv for playbook /opt/contrail/server_manager/ansible/playbooks/ubuntu16newton30/playbooks/site.yml"
"2017-10-29 23:26:47,002-INFO-sm_ansible_playbook.py:188-run_playbook(): Created inventory /opt/contrail/server_manager/ansible/playbooks/ubuntu16newton30/playbooks/inventory/cluster-vcenter-compute.inv for playbook /opt/contrail/server_manager/ansible/playbooks/ubuntu16newton30/playbooks/site.yml"
"2017-10-29 23:26:47,026-INFO-sm_ansible_playbook.py:200-run_playbook(): Starting playbook /opt/contrail/server_manager/ansible/playbooks/ubuntu16newton30/playbooks/site.yml"
"2017-10-29 23:26:47,028-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.10&state=contrail-deploy"
"2017-10-29 23:26:47,031-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s31 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,059-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,063-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.11&state=contrail-deploy"
"2017-10-29 23:26:47,066-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s30 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,093-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,096-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.12&state=contrail-deploy"
"2017-10-29 23:26:47,099-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s29 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,129-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,132-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.18&state=contrail-deploy"
"2017-10-29 23:26:47,135-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s23 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,164-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,167-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.19&state=contrail-deploy"
"2017-10-29 23:26:47,169-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data contrailvm-5a10s27 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,199-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,202-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.20&state=contrail-deploy"
"2017-10-29 23:26:47,205-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data contrailvm-5a10s25 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,239-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,242-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.21&state=contrail-deploy"
"2017-10-29 23:26:47,245-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data contrailvm-5a10s31 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,278-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:47,282-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.15&state=contrail-deploy"
"2017-10-29 23:26:47,284-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s26 contrail-deploy 2017_10_29__23_26_47"
"2017-10-29 23:26:47,314-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:26:59,649-DEBUG-server_mgr_mon_base_plugin.py:779-gevent_puppet_agent_action(): Going to stop the puppet agent on the server 5a10s30"
"2017-10-29 23:26:59,795-INFO-server_mgr_ssh_client.py:65-connect(): CONNECT SUCCESS: Host => 10.87.36.11, option => key"
"2017-10-29 23:26:59,921-DEBUG-server_mgr_mon_base_plugin.py:786-gevent_puppet_agent_action(): OP is ('Ubuntu', '16.04', 'xenial')
"
"2017-10-29 23:26:59,921-DEBUG-server_mgr_mon_base_plugin.py:797-gevent_puppet_agent_action(): PUPPET START Command is sed -i 's/START=.*$/START=yes/' /etc/default/puppet && /usr/bin/puppet resource service puppet ensure=running enable=true "
"2017-10-29 23:27:01,334-DEBUG-server_mgr_mon_base_plugin.py:812-gevent_puppet_agent_action(): Successfully stopped the puppet agent on the server 5a10s30"
"2017-10-29 23:27:01,336-DEBUG-server_mgr_mon_base_plugin.py:825-gevent_puppet_agent_action(): Still trying to stop the puppet agent in the server 5a10s30, try 1"
"2017-10-29 23:27:03,267-DEBUG-server_mgr_mon_base_plugin.py:779-gevent_puppet_agent_action(): Going to stop the puppet agent on the server 5a10s29"
"2017-10-29 23:27:03,382-INFO-server_mgr_ssh_client.py:65-connect(): CONNECT SUCCESS: Host => 10.87.36.12, option => key"
"2017-10-29 23:27:03,472-DEBUG-server_mgr_mon_base_plugin.py:786-gevent_puppet_agent_action(): OP is ('Ubuntu', '16.04', 'xenial')
"
"2017-10-29 23:27:03,472-DEBUG-server_mgr_mon_base_plugin.py:797-gevent_puppet_agent_action(): PUPPET START Command is sed -i 's/START=.*$/START=yes/' /etc/default/puppet && /usr/bin/puppet resource service puppet ensure=running enable=true "
"2017-10-29 23:27:04,973-DEBUG-server_mgr_mon_base_plugin.py:812-gevent_puppet_agent_action(): Successfully stopped the puppet agent on the server 5a10s29"
"2017-10-29 23:27:04,977-DEBUG-server_mgr_mon_base_plugin.py:825-gevent_puppet_agent_action(): Still trying to stop the puppet agent in the server 5a10s29, try 1"
"2017-10-29 23:27:05,042-INFO-server_mgr_ssh_client.py:65-connect(): CONNECT SUCCESS: Host => 10.87.36.12, option => key"
"2017-10-29 23:27:07,233-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.10&state=provision_failed"
"2017-10-29 23:27:07,235-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s31 provision_failed 2017_10_29__23_27_07"
"2017-10-29 23:27:07,267-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:27:07,270-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.11&state=provision_failed"
"2017-10-29 23:27:07,271-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s30 provision_failed 2017_10_29__23_27_07"
"2017-10-29 23:27:07,301-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:27:07,304-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.12&state=provision_failed"
"2017-10-29 23:27:07,306-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s29 provision_failed 2017_10_29__23_27_07"
"2017-10-29 23:27:07,336-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:27:07,339-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.18&state=provision_failed"
"2017-10-29 23:27:07,342-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data 5a10s23 provision_failed 2017_10_29__23_27_07"
"2017-10-29 23:27:07,374-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:27:07,377-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.19&state=provision_failed"
"2017-10-29 23:27:07,380-DEBUG-server_mgr_status.py:134-put_ansible_status(): Server status Data contrailvm-5a10s27 provision_failed 2017_10_29__23_27_07"
"2017-10-29 23:27:07,414-DEBUG-server_mgr_status.py:138-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-10-29 23:27:07,418-INFO-sm_ansible_utils.py:489-send_REST_request(): Sending post request to http://10.87.36.10:9002/ansible_status?server_id=10.87.36.20&state=provision_failed"

>>>>>>> ansible logs that is missing above

TASK [vcenter : Save VM dvs pvlan fix task results {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/922776341200.9937', u'ansible_job_id': u'922776341200.9937', 'changed': True}] ***
ok: [localhost]

TASK [vcenter : Wait for VM DVS-PG pvlan fix task] *******************************************************************************
failed: [localhost] (item={u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/502802295086.9782', u'ansible_job_id': u'502802295086.9782', 'changed': True}) => {"ansible_job_id": "502802295086.9782", "attempts": 1, "changed": true, "cmd": "python /opt/contrail/server_manager/manage_dvs_pg.py --host \"10.87.131.15\" --port 443 --user \"<email address hidden>\" --password \"Contrail123!\" --dvs_name \"Sanity-dvswitch-1\" --dv_pg_name \"Sanity-dvportgroup-1\" --num_ports 1800", "delta": "0:00:00.445704", "end": "2017-10-30 13:19:17.454267", "failed": true, "finished": 1, "item": {"ansible_job_id": "502802295086.9782", "changed": true, "finished": 0, "results_file": "/root/.ansible_async/502802295086.9782", "started": 1}, "rc": 1, "start": "2017-10-30 13:19:17.008563", "stderr": "", "stderr_lines": [], "stdout": "Unable to connect to 10.87.131.15", "stdout_lines": ["Unable to connect to 10.87.131.15"]}
[WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin
(</opt/contrail/server_manager/ansible/plugins/sm_ansible_callback.CallbackModule object at 0x7f2c78d64fd0>): 'CallbackModule'
object has no attribute 'logger'

[WARNING]: Failure using method (v2_runner_on_failed) in callback plugin
(</opt/contrail/server_manager/ansible/plugins/sm_ansible_callback.CallbackModule object at 0x7f2c78d64fd0>): 'CallbackModule'
object has no attribute 'logger'

Tags:

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-31: [Review update] master

Review in progress for https://review.opencontrail.org/37017
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-31: [Review update] R4.1

Review in progress for https://review.opencontrail.org/37018
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-11-01: [Review update] master

Review in progress for https://review.opencontrail.org/37017
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-11-01: [Review update] R4.1

Review in progress for https://review.opencontrail.org/37018
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-11-02: A change has been merged

Reviewed: https://review.opencontrail.org/37018
Committed: http://github.com/Juniper/contrail-server-manager/commit/d976e212306fee2e3aacc4a6a5386088bcad938c
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit d976e212306fee2e3aacc4a6a5386088bcad938c
Author: Ramprakash Ram Mohan <email address hidden>
Date: Tue Oct 31 08:25:05 2017 -0700

SM logging should work even when cluster_id is not found in inventory

SM Logging infra requires the cluster_id for logging cluster specific logging.
For this purpose, the cluster_id is inserted into the inventory. But for ansible
plays that involve only "localhost", the inv.get_hosts() function returns an
empty list. So the host specific variable retrieval fails. Now always using the
first entry of the host_vars dictionary to retrieve the cluster_id.

Also, adding some defensive checks for cluster_id in the PlayLogger class.

Change-Id: I846ba490fe0f1129181eae348721a1df6d35d6af
Closes-bug: #1728751

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-11-02:

#10

Reviewed: https://review.opencontrail.org/37017
Committed: http://github.com/Juniper/contrail-server-manager/commit/b53a44015ad303a5914c29828ac1467350b9b9dd
Submitter: Zuul (<email address hidden>)
Branch: master

commit b53a44015ad303a5914c29828ac1467350b9b9dd
Author: Ramprakash Ram Mohan <email address hidden>
Date: Tue Oct 31 08:25:05 2017 -0700

SM logging should work even when cluster_id is not found in inventory

Also, adding some defensive checks for cluster_id in the PlayLogger class.

Change-Id: I846ba490fe0f1129181eae348721a1df6d35d6af
Closes-bug: #1728751

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.