Juniper Openstack

Server Manager :: R4.0.1.0 build 23 :: SM provision stuck at post provisioning task for multi node setup on non-openstack role node.

Bug #1714099 reported by Ritam Gangopadhyay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Ramprakash R
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Fix Committed
High
Ramprakash R
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

SM IP :: 10.204.216.112

root@nodek7-vm3:~# server-manager status server --cluster_id ocata_cluster

+---------+--------------------------------+----------------+-------------------+
| id | status | ip_address | mac_address |
+---------+--------------------------------+----------------+-------------------+
| nodem16 | provision_completed | 10.204.216.105 | 0C:C4:7A:DC:42:92 |
| nodem17 | openstack_post-deploy-contrail | 10.204.216.106 | 0C:C4:7A:DC:42:D8 |
| nodem18 | openstack_post-deploy-contrail | 10.204.216.107 | 0C:C4:7A:DC:42:D4 |
| nodea10 | openstack_post-deploy-contrail | 10.204.216.48 | 00:25:90:A5:3B:1A |
| nodem19 | openstack_post-deploy-contrail | 10.204.216.108 | 0C:C4:7A:DC:42:D6 |
| nodem20 | openstack_post-deploy-contrail | 10.204.216.109 | 0C:C4:7A:DC:44:42 |
+---------+--------------------------------+----------------+-------------------+
root@nodek7-vm3:~#

Provisioning does not proceed after the task

TASK [Running nova simple cell setup]"

and gets stuck there. So though provisioning is complete and it s just installing post installation utilities the cli gives an idea that it's stuck at openstack_post-deploy-contrail and never moves to provision_completed state

"2017-08-31 02:17:44,568-INFO-sm_ansible_callback.py:43-append(): PLAY [Setup ceph compute node]"
"2017-08-31 02:17:44,571-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.105&state=provision_completed"
"2017-08-31 02:17:44,581-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem16 provision_completed 2017_08_31__02_17_44"
"2017-08-31 02:17:44,673-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:44,678-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.105] : ok: 111 changed: 7 unreachable: 0 skipped: 189 failed: 0"
"2017-08-31 02:17:44,679-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.106&state=provision_completed"
"2017-08-31 02:17:44,682-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem17 provision_completed 2017_08_31__02_17_44"
"2017-08-31 02:17:44,781-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:44,784-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.106] : ok: 111 changed: 6 unreachable: 0 skipped: 189 failed: 0"
"2017-08-31 02:17:44,785-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.107&state=provision_completed"
"2017-08-31 02:17:44,788-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem18 provision_completed 2017_08_31__02_17_44"
"2017-08-31 02:17:44,855-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:44,858-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.107] : ok: 76 changed: 1 unreachable: 0 skipped: 185 failed: 0"
"2017-08-31 02:17:44,859-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.108&state=provision_completed"
"2017-08-31 02:17:44,861-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem19 provision_completed 2017_08_31__02_17_44"
"2017-08-31 02:17:44,931-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:44,933-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.108] : ok: 108 changed: 10 unreachable: 0 skipped: 280 failed: 0"
"2017-08-31 02:17:44,934-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.109&state=provision_completed"
"2017-08-31 02:17:44,936-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem20 provision_completed 2017_08_31__02_17_44"
"2017-08-31 02:17:45,013-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,016-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.109] : ok: 108 changed: 10 unreachable: 0 skipped: 280 failed: 0"
"2017-08-31 02:17:45,017-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.48&state=provision_completed"
"2017-08-31 02:17:45,019-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodea10 provision_completed 2017_08_31__02_17_45"
"2017-08-31 02:17:45,089-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,091-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.48] : ok: 67 changed: 1 unreachable: 0 skipped: 191 failed: 0"
"2017-08-31 02:17:45,094-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=localhost&state=provision_completed"
"2017-08-31 02:17:45,098-DEBUG-server_mgr_status.py:111-put_ansible_status(): Server status Data provision_completed localhost 2017_08_31__02_17_45"
"2017-08-31 02:17:45,100-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [localhost] : ok: 1 changed: 0 unreachable: 0 skipped: 24 failed: 0"
"2017-08-31 02:17:45,102-INFO-sm_ansible_playbook.py:666-create_kolla_param_files(): Changing globals and passwords files"
"2017-08-31 02:17:45,103-INFO-sm_ansible_playbook.py:672-create_kolla_param_files(): Creating /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/../etc/kolla/passwords.yml"
"2017-08-31 02:17:45,124-INFO-sm_ansible_playbook.py:687-create_kolla_param_files(): Creating /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/../etc/kolla/globals.yml"
"2017-08-31 02:17:45,169-INFO-sm_ansible_playbook.py:734-run_playbook(): Creating inventory /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/inventory/ocata_cluster_kolla.inv for playbook /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/post-deploy-contrail.yml"
"2017-08-31 02:17:45,171-INFO-sm_ansible_playbook.py:737-run_playbook(): Created inventory /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/inventory/ocata_cluster_kolla.inv for playbook /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/post-deploy-contrail.yml"
"2017-08-31 02:17:45,377-INFO-sm_ansible_playbook.py:749-run_playbook(): Starting playbook /opt/contrail/server_manager/ansible/playbooks/contrail_23/kolla-ansible/ansible/post-deploy-contrail.yml"
"2017-08-31 02:17:45,379-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.105&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,381-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem16 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,489-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,492-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.106&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,494-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem17 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,572-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,574-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.107&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,577-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem18 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,647-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,649-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.108&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,652-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem19 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,722-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,729-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.109&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,731-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem20 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,797-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:45,799-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.48&state=openstack_post-deploy-contrail"
"2017-08-31 02:17:45,801-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodea10 openstack_post-deploy-contrail 2017_08_31__02_17_45"
"2017-08-31 02:17:45,880-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
Ansible callback init - smgr_ip: 10.204.216.112"2017-08-31 02:17:45,927-INFO-sm_ansible_callback.py:43-append(): PLAY [Post-deploy playbook for contrail]"
"2017-08-31 02:17:45,932-INFO-sm_ansible_callback.py:43-append(): TASK [Gathering Facts]"
"2017-08-31 02:17:46,736-INFO-sm_ansible_callback.py:43-append(): ok: [10.204.216.105]"
"2017-08-31 02:17:46,742-INFO-sm_ansible_callback.py:43-append(): TASK [Install python-openstack]"
"2017-08-31 02:17:47,536-INFO-sm_ansible_callback.py:43-append(): ok: [10.204.216.105]"
"2017-08-31 02:17:47,547-INFO-sm_ansible_callback.py:43-append(): TASK [Disable the nova-compute on the openstack container]"
"2017-08-31 02:17:49,883-INFO-sm_ansible_callback.py:43-append(): changed: [10.204.216.105]"
"2017-08-31 02:17:49,899-INFO-sm_ansible_callback.py:43-append(): TASK [Waiting for nova-compute service up]"
"2017-08-31 02:17:49,934-INFO-sm_ansible_callback.py:43-append(): skipping: [10.204.216.105]"
"2017-08-31 02:17:49,949-INFO-sm_ansible_callback.py:43-append(): TASK [Running nova simple cell setup]"
"2017-08-31 02:17:49,978-INFO-sm_ansible_callback.py:43-append(): skipping: [10.204.216.105]"
"2017-08-31 02:17:49,986-INFO-sm_ansible_utils.py:463-send_REST_request(): Sending post request to http://10.204.216.112:9002/ansible_status?server_id=10.204.216.105&state=provision_completed"
"2017-08-31 02:17:49,989-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data nodem16 provision_completed 2017_08_31__02_17_49"
"2017-08-31 02:17:50,063-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is ocata_cluster ##########"
"2017-08-31 02:17:50,065-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.204.216.105] : ok: 3 changed: 1 unreachable: 0 skipped: 2 failed: 0"

seems like it's updating the status of all the nodes to openstack post installation after provision completed but working on only the openstack node and then fliping its status to provision completed for other nodes it's not doing that

Ritam Gangopadhyay (ritam)
Changed in juniperopenstack:
assignee: nobody → Ramprakash R (ramprakash)
Jeba Paulaiyan (jebap)
tags: added: sanity
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/35151
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/35152
Submitter: Ramprakash R (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/35151
Committed: http://github.com/Juniper/contrail-server-manager/commit/bc6bad63885899d3ada926f5275cea1ea8fb356b
Submitter: Zuul (<email address hidden>)
Branch: master

commit bc6bad63885899d3ada926f5275cea1ea8fb356b
Author: Ramprakash Ram Mohan <email address hidden>
Date: Thu Aug 31 12:53:11 2017 -0700

Fix for not updating status for the right nodes

1. sm_ansible_playbook.py:
* changed update_status function to take into account
whether the update is for kolla hosts or for contrail hosts
* identified dead code to be removed in a later commit
2. sm_ansible_server.py:
* renamed the bottle route from /run_openstack_playbook to
/run_ansible_playbooks as it is the same route for both openstack and
contrail playbooks
* removed start_contrail_playbook function as it is not used
3. sm_ansible_utils.py:
* Added function to calculated hosts in the kolla inventory
4. server_mgr_main.py:
* renamed the REST endpoint to start ansible provision

Closes-bug: #1714099

Change-Id: I4083548541485864d6a582cf570f861df600a3b3

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35152
Committed: http://github.com/Juniper/contrail-server-manager/commit/d4bfbd3eceb309de38a7944d4b1460611949bd76
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit d4bfbd3eceb309de38a7944d4b1460611949bd76
Author: Ramprakash Ram Mohan <email address hidden>
Date: Thu Aug 31 12:53:11 2017 -0700

Fix for not updating status for the right nodes

1. sm_ansible_playbook.py:
* changed update_status function to take into account
whether the update is for kolla hosts or for contrail hosts
* identified dead code to be removed in a later commit
2. sm_ansible_server.py:
* renamed the bottle route from /run_openstack_playbook to
/run_ansible_playbooks as it is the same route for both openstack and
contrail playbooks
* removed start_contrail_playbook function as it is not used
3. sm_ansible_utils.py:
* Added function to calculated hosts in the kolla inventory
4. server_mgr_main.py:
* renamed the REST endpoint to start ansible provision

Closes-bug: #1714099

Change-Id: I4083548541485864d6a582cf570f861df600a3b3

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.