[docs] Full MySQL & RabbitMQ clusters reassemble may take up to the 20 minutes

[Bogdan Dobrelya]

Steps to reproduce:
1) deploy Ubuntu deploy_neutron_vlan_ha test group (3 controllers, 2 computes)
2) power on destroyed virtual environment after the test group deploy_neutron_vlan was finished w/o errors
3) wait for the full clusters reassemble to finish

After 15 minutes of cluster uptime,

OSTF HA healthcheck reports
[ 1 of 3] [failure] 'Check data replication over mysql' (0.5401 s) Mysql node detection failed Please refer to OpenStack logs for more details.
[ 2 of 3] [failure] 'Check amount of tables in databases is the same on each node' (0.5767 s) Can list tables Please refer to OpenStack logs for more details.
[ 3 of 3] [success] 'Check galera environment state' (0.2603 s)

pcs status shows
...
 Clone Set: clone_p_mysql [p_mysql]
     p_mysql (ocf::fuel:mysql-wss): Started node-2 FAILED
     Stopped: [ node-1 node-4 ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Slaves: [ node-1 node-2 node-4 ]
...

Note, It was finished complete reassemble only after 18 minutes of uptime.
Perhaps we should document that expected full cluster reassemble time is ~20 minutes

Used ISO info:
api: '1.0'
astute_sha: 93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c
auth_required: true
build_id: 2015-03-15_22-54-44
build_number: '200'
feature_groups:
- mirantis
fuellib_sha: 553cb0cffa40a5f57313f962b6ec6a9bd89306ba
fuelmain_sha: c97fd8a789645bda48d06da224f994f8b52d82f5
nailgun_sha: 713e6684f9f54e29acfe6b8ebf641b9de2292628
ostf_sha: e86c961ceacfa5a8398b6cbda7b70a5f06afb476
production: docker
python-fuelclient_sha: cc64fff91fb0d78e5a02e7b93ceff224296f84fb
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c
      build_id: 2015-03-15_22-54-44
      build_number: '200'
      feature_groups:
      - mirantis
      fuellib_sha: 553cb0cffa40a5f57313f962b6ec6a9bd89306ba
      fuelmain_sha: c97fd8a789645bda48d06da224f994f8b52d82f5
      nailgun_sha: 713e6684f9f54e29acfe6b8ebf641b9de2292628
      ostf_sha: e86c961ceacfa5a8398b6cbda7b70a5f06afb476
      production: docker
      python-fuelclient_sha: cc64fff91fb0d78e5a02e7b93ceff224296f84fb
      release: '6.1'

[Bogdan Dobrelya (bogdando)]

Logs are here http://goo.gl/wmEEng

[Bogdan Dobrelya (bogdando)]

The solution is yet unknown but the documentation update should be provided, hence triaged

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/184674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184674
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a2a146972dc63208280920ff1cf9321a6300171b
Submitter: Jenkins
Branch: master

commit a2a146972dc63208280920ff1cf9321a6300171b
Author: Vladimir Kuklin <email address hidden>
Date: Thu May 21 05:04:57 2015 +0300

    Set mnesia_table_loading_timeout to 10 seconds

    This commit sets mnesia_table_loading_timeout to
    10 seconds thus making rabbitmq cluster failover
    process faster. This option was initially suggested
    by Michael Klishin (RabbitMQ developer)

    Change-Id: I8ff6388cdd785404ea3659584b20b9e977a1c253
    Related-bug: #1455761
    Related-bug: #1432603

[Bogdan Dobrelya (bogdando)]

Marking invalid as the rabbitmq cluster reassemble time was drastically improved in the 7.0.