Deployment Failed MongoDB: Command exceeded timeout

[Tyler Wilson]

{"build_id": "2014-10-15_17-43-08", "ostf_sha": "de177931b53fbe9655502b73d03910b8118e25f1", "build_number": "167", "auth_required": true, "api": "1.0", "nailgun_sha": "d9facd6a32293da786b738d1a9b1459e36aa3006", "production": "docker", "fuelmain_sha": "18b8264c17782c4dbb77412d3d4fe256a2083d7d", "astute_sha": "c3e7c7a18528cf9acca48021488a93dff74f5c97", "feature_groups": ["experimental"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-10-15_17-43-08", "ostf_sha": "de177931b53fbe9655502b73d03910b8118e25f1", "build_number": "167", "api": "1.0", "nailgun_sha": "d9facd6a32293da786b738d1a9b1459e36aa3006", "production": "docker", "fuelmain_sha": "18b8264c17782c4dbb77412d3d4fe256a2083d7d", "astute_sha": "c3e7c7a18528cf9acca48021488a93dff74f5c97", "feature_groups": ["experimental"], "release": "6.0", "fuellib_sha": "b3f6943326dac065464555a320ed4b2d4bdbb699"}}}, "fuellib_sha": "b3f6943326dac065464555a320ed4b2d4bdbb699"}

1. Setup Env with Ubuntu HA + Ceph + Neutron + GRE
2. Add 3x Controllers + MongoDB on the same nodes
3. Add 3x Compute Nodes + Ceph OSD
4. Observe the following timeout on deployment

2014-10-17 21:47:40 ERR
 (/Stage[main]/Mongodb::Replset::Install/Exec[wait_for_server]/returns) change from notrun to 0 failed: Command exceeded timeout
2014-10-17 21:47:40 ERR
 /usr/bin/puppet:4
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `evaluate'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `thinmark'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `evaluate'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:193:in `sync'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:98:in `sync'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-10-17 21:47:40 ERR
 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `each'
2014-10-17 21:47:40 ERR
 Command exceeded timeout

[Timur Nurlygayanov (tnurlygayanov)]

This is not bug in Murano.

We can see the timeout of the primary controller with Mongo Data Base deploy. This is issue in puppet manifests or with default timeouts, looks like fuel-library team can solve this issue.

[Mike Scherbakov (mihgen)]

Matt, are you working on the bug?
If not, let's assign it back to fuel-library and find free a hands to work on it.

[Matthew Mosesohn (raytrac3r)]

I'm working on this. I'm having a bit of difficulty reproducing due to other HA failures when you combine MongoDB on master.

[Bogdan Dobrelya (bogdando)]

Please do not change the status to 'in progress', if there is a team assigned to it

[Oleksiy Molchanov (omolchanov)]

Cannot reproduce on the latest build.

I suppose that the issue was related to Exec['wait_for_server'] that was improved in patch for 1381826.