MU installation using "fuel2 update" failed on "deployment_graph_task_name"=>"cluster"

[Artem Hrechanychenko]

Detailed bug description:
 9.1 MU installation to 9.0 cluster failed on "deployment_graph_task_name"=>"cluster"

Steps to reproduce:
1) Deploy any 9.0 cluster
2) configure additional yum repo:http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-latest/x86_64/
3)yum -y update fuel-nailgun fuel-utils python-fuelclient rubygem-astute
  bash /usr/share/fuel-utils/update-master-node.sh
4) yum install -y python-cudet
5) fuel2 update --env 1 install (--restart-rabbitmq|--restart-mysql)

Expected results:
MU successfully installed by deployment task

Actual result:
 [12053] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=> {"task_uuid"=>"07e238a3-de4c-4e24-bce2-18cc5ea79f73", "nodes"=> [{"uid"=>"2", "progress"=>100, "status"=>"error", "error_type"=>"deploy", "deployment_graph_task_name"=>"cluster", "task_status"=>"error", "custom"=> {:time=> {"anchor"=>0.000358116, "augeas"=>0.07585699, "config_retrieval"=>2.991711518, "file"=>0.15906029500000002, "filebucket"=>0.000132497, "notify"=>0.003919227, "package"=>0.000852258, "pcmk_nodes"=>1815.707277099, "pcmk_property"=>1.859603124, "schedule"=>0.000956795, "service"=>5.327626973, "total"=>1826.127354892, "last_run"=>1472330926}, :resources=> {"changed_resources"=> "Notify[restarting services],Service[pacemaker]", "failed_resources"=>"Pcmk_nodes[pacemaker]", "failed"=>1, "changed"=>2, "total"=>52, "restarted"=>1, "out_of_sync"=>3, "failed_to_restart"=>0, "scheduled"=>0, "skipped"=>0}, :changes=>{"total"=>2}, :events=>{"failure"=>1, "success"=>2, "total"=>3}, :version=>{"config"=>1472329097, "puppet"=>"3.8.5"}, :raw_report=>nil, :status=>"running", :running=>1, :enabled=>1, :idling=>0, :stopped=>0, :lastrun=>1472330926, :runtime=>1766, :output=>"Currently running; last completed run 1766 seconds ago"}, "error_msg"=>"Task cluster failed on node 2"}]}}

[Artem Hrechanychenko (agrechanichenko)]

shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 python-packetary-9.0.0-1.mos144.noarch
 fuel-misc-9.0.0-1.mos8551.noarch
 rubygem-astute-9.0.0-1.mos761.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuelmenu-9.0.0-1.mos275.noarch
 fuel-utils-9.0.0-1.mos8551.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8790.noarch
 fuel-agent-9.0.0-1.mos289.noarch
 fuel-notify-9.0.0-1.mos8551.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-nailgun-9.0.0-1.mos8790.noarch
 fuel-bootstrap-cli-9.0.0-1.mos289.noarch
 fuel-ostf-9.0.0-1.mos940.noarch
 nailgun-mcagents-9.0.0-1.mos761.noarch
 fuel-migrate-9.0.0-1.mos8551.noarch
 python-fuelclient-9.0.0-1.mos341.noarch
 fuel-9.0.0-1.mos6352.noarch
 fuel-ui-9.0.0-1.mos2777.noarch
 fuel-release-9.0.0-1.mos6352.noarch
 fuel-mirror-9.0.0-1.mos144.noarch
 fuel-library9.0-9.0.0-1.mos8551.noarch
 fuel-openstack-metadata-9.0.0-1.mos8790.noarch
 fuel-setup-9.0.0-1.mos6352.noarch

[Ivan Berezovskiy (iberezovskiy)]

If we run cluster task on one controller and pacemaker isn't alive (dead or not fully up yet) on other 2 controller, the cluster tasks hangs on:
/usr/sbin/crm_attribute -q --type crm_config --query --name dc-version

If pacemaker service become up on other nodes, the cluster task still hangs on this command. The issue I see here that cibadmin query isn't refreshed before `/usr/sbin/crm_attribute -q --type crm_config --query --name dc-version` command. That's why I think we need to update it every time.

https://review.fuel-infra.org/#/c/25620/ -fix for puppet-pacemaker module.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/363017

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/363032

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/363033

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/363037

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Berezovskiy (<email address hidden>) on branch: master
Review: https://review.openstack.org/363032
Reason: https://review.openstack.org/#/c/363017/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Berezovskiy (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/363033
Reason: https://review.openstack.org/#/c/363017/

[Ivan Berezovskiy (iberezovskiy)]

Patches for both branches:
https://review.openstack.org/#/c/363037/
https://review.openstack.org/#/c/363017/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363017
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a95370bedc1725f226e09e1206a83a9a282790a1
Submitter: Jenkins
Branch: master

commit a95370bedc1725f226e09e1206a83a9a282790a1
Author: Dmitry Ilyin <email address hidden>
Date: Tue Aug 30 11:31:17 2016 -0500

    Push pacemaker version to 1.0.8

    Related-bug: #1617964

    Change-Id: I80101661279b4c0455a84f61ea83d2974a811f22

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363037
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9798f2a61c3b6a27304b559bb27f19395f33f55b
Submitter: Jenkins
Branch: stable/mitaka

commit 9798f2a61c3b6a27304b559bb27f19395f33f55b
Author: Dmitry Ilyin <email address hidden>
Date: Tue Aug 30 11:31:17 2016 -0500

    Push pacemaker version to 1.0.8

    Cherry picked from: a95370bedc1725f226e09e1206a83a9a282790a1
    Change-Id: I80101661279b4c0455a84f61ea83d2974a811f22
    Related-bug: #1617964