Environment deployment fails if controller node is replaced (old node is removed and new one is added):
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 1 took 00:00:22
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 2 took 00:00:22
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 6 took 00:00:22
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "nodes"=> [{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}, {"uid"=>"2", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}, {"uid"=>"6", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}], "error"=> "Failed to execute hook 'dump_rabbitmq_definitions' Puppet run failed. Check puppet logs for details"}}
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "nodes"=> [{"uid"=>"3", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"5", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}]}}
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "status"=>"error", "error"=> "Method granular_deploy. Failed to execute hook 'dump_rabbitmq_definitions' Puppet run failed. Check puppet logs for details --- uids: - '1' - '2' - '6' parameters: puppet_modules: /etc/puppet/modules puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/astute/dump_rabbitmq_definitions.pp timeout: 180 cwd: / priority: 100 fail_on_error: true type: puppet id: dump_rabbitmq_definitions . Inspect Astute logs for the details"}}
node-2 2016-02-02T02:39:09.797343 err: (/Stage[main]/Main/Exec[rabbitmq-dump-definitions]/returns) change from notrun to 0 failed: curl -u nova:ok4dJUpvLZunACcORtB0kiqX http://localhost:15672/api/definitions -o /etc/rabbitmq/definitions.full returned 7 instead of one of [0]
root@node-2:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@messaging-node-2' ...
Error: unable to connect to node 'rabbit@messaging-node-2': nodedown
Segmentation fault
Steps to reproduce:
1. Deploy environment with 3 controllers, 2 computes and 1 compute+cinder nodes
2. Remove 1 compute+cinder node and add 1 controller+cinder node
3. Deploy changes
Expected result: controller is replaced, deployment is successful, environment passes OSTF
Actual result: deployment of controllers fails on dump_rabbitmq_definitions task
Environment deployment fails if controller node is replaced (old node is removed and new one is added):
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_ definitions with status error on node 1 took 00:00:22 definitions with status error on node 2 took 00:00:22 definitions with status error on node 6 took 00:00:22 {"method" =>"deploy_ resp", "args"=> {"task_ uuid"=> "31bd49c1- fb2c-412f- 962a-11381a7b78 1a", "nodes"=> [{"uid"=>"1", "status"=>"error", "error_ type"=> "deploy" , "role"=>"hook", "hook"= >"dump_ rabbitmq_ definitions" , "error_ msg"=>" Puppet run failed. Check puppet logs for details"}, {"uid"=>"2", "status"=>"error", "error_ type"=> "deploy" , "role"=>"hook", "hook"= >"dump_ rabbitmq_ definitions" , "error_ msg"=>" Puppet run failed. Check puppet logs for details"}, {"uid"=>"6", "status"=>"error", "error_ type"=> "deploy" , "role"=>"hook", "hook"= >"dump_ rabbitmq_ definitions" , "error_ msg"=>" Puppet run failed. Check puppet logs for details"}], "error"=> "Failed to execute hook 'dump_rabbitmq_ definitions' Puppet run failed. Check puppet logs for details"}} {"method" =>"deploy_ resp", "args"=> {"task_ uuid"=> "31bd49c1- fb2c-412f- 962a-11381a7b78 1a", "nodes"=> [{"uid"=>"3", "status"=>"error", "role"=>"hook", "error_ type"=> "deploy" }, {"uid"=>"5", "status"=>"error", "role"=>"hook", "error_ type"=> "deploy" }]}} {"method" =>"deploy_ resp", "args"=> {"task_ uuid"=> "31bd49c1- fb2c-412f- 962a-11381a7b78 1a", "status"=>"error", "error"=> "Method granular_deploy. Failed to execute hook 'dump_rabbitmq_ definitions' Puppet run failed. Check puppet logs for details --- uids: - '1' - '2' - '6' parameters: puppet_modules: /etc/puppet/modules puppet_manifest: /etc/puppet/ modules/ osnailyfacter/ modular/ astute/ dump_rabbitmq_ definitions. pp timeout: 180 cwd: / priority: 100 fail_on_error: true type: puppet id: dump_rabbitmq_ definitions . Inspect Astute logs for the details"}}
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:
node-2 2016-02- 02T02:39: 09.797343 err: (/Stage[ main]/Main/ Exec[rabbitmq- dump-definition s]/returns) change from notrun to 0 failed: curl -u nova:ok4dJUpvLZ unACcORtB0kiqX http:// localhost: 15672/api/ definitions -o /etc/rabbitmq/ definitions. full returned 7 instead of one of [0]
root@node-2:~# pcs status | grep -A 2 p_rabbitmq-server p_rabbitmq- server [p_rabbitmq-server] test.domain. local ] test.domain. local node-6. test.domain. local ]
Master/Slave Set: master_
Masters: [ node-1.
Slaves: [ node-2.
root@node-2:~# rabbitmqctl cluster_status messaging- node-2' ... messaging- node-2' : nodedown
Cluster status of node 'rabbit@
Error: unable to connect to node 'rabbit@
Segmentation fault
Steps to reproduce:
1. Deploy environment with 3 controllers, 2 computes and 1 compute+cinder nodes
2. Remove 1 compute+cinder node and add 1 controller+cinder node
3. Deploy changes
Expected result: controller is replaced, deployment is successful, environment passes OSTF
Actual result: deployment of controllers fails on dump_rabbitmq_ definitions task