HA scalability doesn't work

[Nastya Urlapova]

{
build_id: "2014-04-24_15-01-23",
mirantis: "yes",
build_number: "135",
nailgun_sha: "dcaed120b34dd2c8cf817463a243eedfb844f096",
production: "prod",
ostf_sha: "134765fcb5a07dce0cd1bb399b2290c988c3c63b",
fuelmain_sha: "387d2e931ee14d5773f21f1e2860a13b50b37d04",
astute_sha: "6e8fa4cc12968d7b468fc590b2f06bb59bf74511",
release: "5.0",
fuellib_sha: "fa57b98344e7cdd342b7c08a5e692525b102af8b"
}

1) deploy 1xController
2) add 2xControllers or 5xController - redeploy changes

 (/Stage[main]/Openstack::Ha::Mysqld/Exec[wait-for-haproxy-mysql-backend]/returns) change from notrun to 0 failed: echo show stat | socat unix-connect:///var/lib/haproxy/stats stdio | grep -q '^mysqld,BACKEND,.*,UP,' returned 1 instead of one of [0]
2014-04-25 09:28:56 ERR
 /usr/bin/puppet:4
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-04-25 09:28:56 ERR
 echo show stat | socat unix-connect:///var/lib/haproxy/stats stdio | grep -q '^mysqld,BACKEND,.*,UP,' returned 1 instead of one of [0]

[Vladimir Kuklin (vkuklin)]

it looks like the problem is because of corosync restart due to config change because we are using unicast for corosync

[Sergey Vasilenko (xenolog)]

NOTE:
This check by socat should been executed only on the first deploing controller.
And shouldn't while redeploy on any.

[Sergii Golovatiuk (sgolovatiuk)]

Fuel implementation uses static corosync.conf where all nodes are described. In order to have dynamic node configuration there should be another layer of abstraction CMAN or pcsd. CMAN works with corosync 1.x and has been deprecated in corosync 2.x. pcsd has some problems with ubuntu and designed for corosync 2.0 which requires redesign of puppet manifests.

[Mike Scherbakov (mihgen)]

> which requires redesign of puppet manifests.
looks like large change to have. Postponing to 5.1.

[Meg McRoberts (dreidellhasa)]

Added to Known Issues in 5.0 Release Notes.

[Bogdan Dobrelya (bogdando)]

depends on https://blueprints.launchpad.net/fuel/+spec/ha-pacemaker-improvements

[Meg McRoberts (dreidellhasa)]

Marked as "Known Issue" in 5.0.1 Release Notes.

[Dmitry Ilyin (idv1985)]

FIx commited with pacemaker improvements

[Anastasia Palkina (apalkina)]

Verified on ISO #347
"build_id": "2014-07-23_02-01-14",
"ostf_sha": "c1b60d4bcee7cd26823079a86e99f3f65414498e",
"build_number": "347",
"auth_required": false,
"api": "1.0",
"nailgun_sha": "f5775d6b7f5a3853b28096e8c502ace566e7041f",
"production": "docker",
"fuelmain_sha": "74b9200955201fe763526ceb51607592274929cd",
"astute_sha": "fd9b8e3b6f59b2727b1b037054f10e0dd7bd37f1",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "fb0e84c954a33c912584bf35054b60914d2a2360"

[Vladimir Kuklin (vkuklin)]

addressed by fixing https://bugs.launchpad.net/fuel/+bug/1340172