Controller deployment failed on HA (error with mysql show status)

[Alexander Maretskiy]

api: '1.0'
astute_sha: 18a6861ecf96e20ead17782ef4ea8d62e7449c14
auth_required: false
build_id: 2014-07-15_02-19-18
build_number: '320'
feature_groups:
- mirantis
fuellib_sha: b2caaea7b0a39f40ff08240da4dab81aaba6e29c
fuelmain_sha: d12f4ac69a6299130b360e788b098c1f304b28c3
nailgun_sha: 902cbb4ac4995a13dfaedb6a5111f177c1df9ab5
ostf_sha: 9863db951a6e159f4fa6e6861c8331e1af069cf8
production: docker
release: '5.1'

Steps to reproduce:
  * create env:
         - CentOS / HA / Neutron GRE / Cinder, Glance - Default / Sahara, Murano / Ceilometer - NO
         - 3 controllers nodes, 6 compute nodes
 * run deploy

Expected result:
  Successful deployment

Actual result:
  Second controller failed to deploy.
  Its first puppet error is:
  2014-07-15 14:13:48 ERR /usr/bin/mysql -uwsrep_sst -ppassword -Nbe "show status like 'wsrep_local_state_comment'" | /bin/grep -q Synced && sleep 10 returned 1 instead of one of [0]

Extra information:
  [root@node-11 ~]# /usr/bin/mysql -uwsrep_sst -ppassword -Nbe "show status like 'wsrep_local_state_comment'"
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)

[Ilya Shakhat (shakhat)]

Part of puppet.log related to the error:

Tue Jul 15 13:05:54 +0000 2014 /Stage[main]/Galera/File[/etc/mysql/conf.d/wsrep.cnf]/ensure (notice): created
Tue Jul 15 13:05:54 +0000 2014 Puppet (notice): Installing package MySQL-server-wsrep from absent to 5.6.16_wsrep_25.5.mira1-1.mira2
Tue Jul 15 13:06:21 +0000 2014 /Stage[main]/Galera/Package[MySQL-server]/ensure (notice): created
Tue Jul 15 13:06:21 +0000 2014 /Stage[main]/Galera/File[/etc/init.d/mysql]/mode (notice): mode changed '0755' to '0644'
Tue Jul 15 13:06:21 +0000 2014 /Stage[main]/Galera/File[mysql-wss-ocf]/ensure (notice): defined content as '{md5}480e634544018f9007bb8dc
33b25fadb'
Tue Jul 15 13:07:08 +0000 2014 /Stage[main]/Galera/Service[mysql]/ensure (notice): ensure changed 'stopped' to 'running'
Tue Jul 15 13:08:40 +0000 2014 /Stage[main]/Galera/Exec[wait-initial-sync] (notice): Triggered 'refresh' from 1 events
Tue Jul 15 13:13:48 +0000 2014 /Stage[main]/Galera/Exec[wait-for-synced-state]/returns (notice): ERROR 2002 (HY000): Can't connect to lo
cal MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Tue Jul 15 13:13:48 +0000 2014 Puppet (err): /usr/bin/mysql -uwsrep_sst -ppassword -Nbe "show status like 'wsrep_local_state_comment'" |
 /bin/grep -q Synced && sleep 10 returned 1 instead of one of [0]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/107072

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/107072
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d6636dd02bde07993dcd875afd8c66e61e9ea31a
Submitter: Jenkins
Branch: master

commit d6636dd02bde07993dcd875afd8c66e61e9ea31a
Author: Dmitry Ilyin <email address hidden>
Date: Tue Jul 15 18:40:17 2014 +0400

    Make all nodes appear in mysql-wss config

    * New Galera requires all nodes in the config
    * Remove unused galera_final_config class

    Closes-Bug: 1342128

    Change-Id: I47eabb0e5272bb8fdeac5fcd8d8173d9ce106202

[Bogdan Dobrelya (bogdando)]

Note that this patchset discarded all wsrep config on scale changes were in master branch