Environment deployment fails if controller node is replaced (old node is removed and new one is added):
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 1 took 00:00:22
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 2 took 00:00:22
2016-02-02 02:39:16 DEBUG [796] Task time summary: dump_rabbitmq_definitions with status error on node 6 took 00:00:22
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "nodes"=> [{"uid"=>"1", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}, {"uid"=>"2", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}, {"uid"=>"6", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"dump_rabbitmq_definitions", "error_msg"=>"Puppet run failed. Check puppet logs for details"}], "error"=> "Failed to execute hook 'dump_rabbitmq_definitions' Puppet run failed. Check puppet logs for details"}}
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "nodes"=> [{"uid"=>"3", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"5", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}]}}
2016-02-02 02:39:16 INFO [796] Casting message to Nailgun:{"method"=>"deploy_resp", "args"=> {"task_uuid"=>"31bd49c1-fb2c-412f-962a-11381a7b781a", "status"=>"error", "error"=> "Method granular_deploy. Failed to execute hook 'dump_rabbitmq_definitions' Puppet run failed. Check puppet logs for details --- uids: - '1' - '2' - '6' parameters: puppet_modules: /etc/puppet/modules puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/astute/dump_rabbitmq_definitions.pp timeout: 180 cwd: / priority: 100 fail_on_error: true type: puppet id: dump_rabbitmq_definitions . Inspect Astute logs for the details"}}
node-2 2016-02-02T02:39:09.797343 err: (/Stage[main]/Main/Exec[rabbitmq-dump-definitions]/returns) change from notrun to 0 failed: curl -u nova:ok4dJUpvLZunACcORtB0kiqX http://localhost:15672/api/definitions -o /etc/rabbitmq/definitions.full returned 7 instead of one of [0]
root@node-2:~# pcs status | grep -A 2 p_rabbitmq-server
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Masters: [ node-1.test.domain.local ]
Slaves: [ node-2.test.domain.local node-6.test.domain.local ]
root@node-2:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@messaging-node-2' ...
Error: unable to connect to node 'rabbit@messaging-node-2': nodedown
Segmentation fault
Steps to reproduce:
1. Deploy environment with 3 controllers, 2 computes and 1 compute+cinder nodes
2. Remove 1 controller node and add 1 controller+cinder node
3. Deploy changes
Expected result: controller is replaced, deployment is successful, environment passes OSTF
Actual result: deployment of controllers fails on dump_rabbitmq_definitions task
The rabbitmq-server process definitely wasn't running on node-2. It looks like it went down around 1 minute after node-4 was fenced (00:46). I don't see any rabbitmq server logs in the snapshot though so I can't tell what happened to node-2 at that time. lrmd.log (http://