Provider pacemaker is not functional on this host

[Sergey Galkin]

Snapshot attached to https://bugs.launchpad.net/fuel/+bug/1389640

api: '1.0'
astute_sha: c72dac7b31646fbedbfc56a2a87676c6d5713fcf
auth_required: true
build_id: 2014-11-02_21-27-58
build_number: '69'
feature_groups:
- mirantis
fuellib_sha: 45ad9b42666d7e3e14ab9af2911808e6c8806842
fuelmain_sha: ac3ba5f5c6073b7776ec69fc3cb4dd3c56df36c5
nailgun_sha: 35946b1f225c984f11915ba8e985584160f0b129
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'

Steps to reproduce:
1. Try to Install 100 nodes cluster with 3 controller in HA + 97 compute with cinder iscsi and neutron gre

Deployment has failed with message:
"Deployment has failed. Method deploy. Disabling the upload of disk image because glance was not installed properly.
Inspect Astute logs for the details"

In the puppet error logs on the failed controller controller_1:

2014-11-05 11:14:02 ERR
 /usr/bin/puppet:4
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:67:in `evaluate'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:167:in `prefetch_if_necessary'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `prefetch'
2014-11-05 11:14:02 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:61:in `prefetch'
2014-11-05 11:14:02 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/cs_property/crm.rb:19:in `instances'
2014-11-05 11:14:02 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:48:in `block_until_ready'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/1.8/timeout.rb:64:in `block_until_ready'
2014-11-05 11:14:02 ERR
 Could not run: execution expired
2014-11-05 11:11:10 ERR
 /usr/bin/puppet:4
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:67:in `evaluate'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:167:in `prefetch_if_necessary'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `prefetch'
2014-11-05 11:11:10 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:61:in `prefetch'
2014-11-05 11:11:10 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/cs_property/crm.rb:19:in `instances'
2014-11-05 11:11:10 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:48:in `block_until_ready'
2014-11-05 11:11:10 ERR
 /usr/lib/ruby/1.8/timeout.rb:64:in `block_until_ready'
2014-11-05 11:11:10 ERR
 Could not run: execution expired
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for neutron_router_interface
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for neutron_router
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for neutron_subnet
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for neutron_network
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for cs_order
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for cs_colocation
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for cs_location
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for cs_resource
2014-11-05 11:08:17 ERR
 Could not find a suitable provider for cs_property
2014-11-05 11:08:17 ERR
 (/Stage[main]/Neutron::Agents::Metadata/Service[neutron-metadata]) Provider pacemaker is not functional on this host
2014-11-05 11:08:17 ERR
 (/Stage[main]/Neutron::Agents::L3/Service[neutron-l3]) Provider pacemaker is not functional on this host
2014-11-05 11:08:17 ERR
 (/Stage[main]/Heat::Engine/Service[heat-engine_service]) Provider pacemaker is not functional on this host
2014-11-05 11:08:17 ERR
 (/Stage[main]/Neutron::Agents::Dhcp/Service[neutron-dhcp-service]) Provider pacemaker is not functional on this host
2014-11-05 11:08:17 ERR
 (/Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]) Provider pacemaker is not functional on this host
2014-11-05 11:08:09 ERR
 (/Stage[main]/Cluster::Haproxy_ocf/Service[p_haproxy]) Provider pacemaker is not functional on this host
2014-11-05 11:08:00 ERR
 (/Stage[main]/Galera/Service[mysql-service]) Provider pacemaker is not functional on this host
2014-11-05 11:08:00 ERR
 (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) Provider pacemaker is not functional on this host

[Bogdan Dobrelya (bogdando)]

Superseded by bp https://blueprints.launchpad.net/fuel/+spec/pacemaker-improvements
New pacemaker provider should fix it as well

[Bogdan Dobrelya (bogdando)]

Specifically, https://review.openstack.org/#/c/129351/ should fix that. I suggest to retest this case once new provider merged

[Bogdan Dobrelya (bogdando)]

2014-11-05 11:14:02 ERR
 /etc/puppet/modules/corosync/lib/puppet/provider/corosync.rb:48:in `block_until_ready'
2014-11-05 11:14:02 ERR
 /usr/lib/ruby/1.8/timeout.rb:64:in `block_until_ready'
2014-11-05 11:14:02 ERR
 Could not run: execution expired

means what corosync service was not ready to accept requests for default 120 seconds

[Bogdan Dobrelya (bogdando)]

yes, I can see from logs that Corosync failed to up and run:
./node-32.domain.tld/puppet-apply.log:2014-11-05T11:09:10.703074+00:00 info: (/Stage[corosync_setup]/Corosync/Package[corosync]) Evaluated in 0.00 sec
onds
./node-32.domain.tld/puppet-apply.log:2014-11-05T11:09:11.001988+00:00 debug: (Puppet::Type::Cs_property::ProviderCrm) Corosync not ready, retrying
...
./node-32.domain.tld/puppet-apply.log:2014-11-05T11:11:08.988330+00:00 debug: (Puppet::Type::Cs_property::ProviderCrm) Corosync not ready, retrying

caused this failure

[Dmitry Borodaenko (angdraug)]

The right status for a bug superceded by a blueprint is Invalid. Won't Fix means we're not going to address it, Invalid means it's a feature request, not a bug.