[Detached services] Mysql replication tests failed on ready cluster after node additional

[Tatyanka]

Add and delete keystone node

Scenario:
1. Revert snapshot separate_keystone_service
2. Add one keystone node and re-deploy cluster
3. Run network verification
4. Run OSTF ==================================================> FAIL HERE
5. Check hiera hosts are the same for
different group of roles
6. Delete one keystone node
7. Run network verification
8. Run ostf
9. Check hiera hosts are the same for
different group of roles

Actual:
OSTF mysql tests are fails:
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|--------------------------------------------------|---------|--------------|-------------------|------------------------------------------|---------------|--------|---------
3 | ready | slave-05_standalone-database_standalone-keystone | 1 | 10.109.10.7 | 64:91:39:d4:bf:1a | standalone-database, standalone-keystone | | True | 1
4 | ready | slave-04_standalone-database_standalone-keystone | 1 | 10.109.10.6 | 64:06:19:87:54:96 | standalone-database, standalone-keystone | | True | 1
6 | ready | slave-06_standalone-database_standalone-keystone | 1 | 10.109.10.8 | 64:6d:76:78:36:5a | standalone-database, standalone-keystone | | True | 1
9 | ready | slave-09_standalone-keystone | 1 | 10.109.10.11 | 64:92:45:8f:26:b9 | standalone-keystone | | True | 1
7 | ready | slave-07_compute | 1 | 10.109.10.9 | 64:94:ff:a0:9a:48 | compute | | True | 1
8 | ready | slave-08_cinder | 1 | 10.109.10.10 | 64:e0:63:82:4d:fc | cinder | | True | 1
2 | ready | slave-02_controller | 1 | 10.109.10.4 | 64:f7:74:dc:c8:e5 | controller | | True | 1
1 | ready | slave-01_controller | 1 | 10.109.10.3 | 64:ae:71:62:cd:89 | controller | | True | 1
5 | ready | slave-03_controller | 1 | 10.109.10.5 | 64:e7:d2:e2:72:8e | controller | | True | 1
[root@nailgun ~]# fuel health --env 1 --check ha
[ 1 of 7] [success] 'Check state of haproxy backends on controllers' (1.567 s)
[ 2 of 7] [failure] 'Check data replication over mysql' (3.125 s) Can not get data from database node node-4 Please refer to OpenStack logs for more details.
[ 3 of 7] [success] 'Check if amount of tables in databases is the same on each node' (2.739 s)
[ 4 of 7] [failure] 'Check galera environment state' (0.7393 s) Actual value - 3,
[ 5 of 7] [success] 'Check pacemaker status' (1.829 s)
[ 6 of 7] [success] 'RabbitMQ availability' (9.08 s)
[ 7 of 7] [success] 'RabbitMQ replication' (19.7 s)

on node-3:
root@node-3:~# mysql -h localhost -e "SHOW STATUS LIKE 'wsrep_%'" | grep wsrep_connected
wsrep_connected ON
root@node-3:~# mysql -h localhost -e "SHOW STATUS LIKE 'wsrep_%'" | grep wsrep_cluster_size
wsrep_cluster_size 2
root@node-3:~# mysql -h localhost -e "SHOW STATUS LIKE 'wsrep_%'" | grep wsrep_cluster_size
wsrep_cluster_size 2
root@node-3:~# crm_mon -1
Last updated: Fri Feb 12 11:15:26 2016
Last change: Fri Feb 12 03:08:03 2016
Stack: corosync
Current DC: node-3.test.domain.local (3) - partition with quorum
Version: 1.1.12-561c4cf
3 Nodes configured
9 Resources configured

Online: [ node-3.test.domain.local node-4.test.domain.local node-6.test.domain.local ]

 vip__public_service_endpoint (ocf::fuel:ns_IPaddr2): Started node-3.test.domain.local
 vip__service_endpoint (ocf::fuel:ns_IPaddr2): Started node-3.test.domain.local
 vip__database (ocf::fuel:ns_IPaddr2): Started node-3.test.domain.local
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-3.test.domain.local node-4.test.domain.local node-6.test.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-3.test.domain.local node-4.test.domain.local node-6.test.domain.local ]

Failed actions:
    p_mysql_monitor_60000 on node-3.test.domain.local 'unknown error' (1): call=91, status=complete, last-rc-change='Fri Feb 12 03:01:54 2016', queued=0ms, exec=0ms

root@node-3:~#

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "553"
  build_id: "553"
  fuel-nailgun_sha: "ed2e0cde96ae7bc064e689f7409470e69c57772e"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"
[root@nailgun ~]#

[Tatyanka (tatyana-leontovich)]

https://drive.google.com/a/mirantis.com/file/d/0B_tSitrwrgvoampPVk93aFR4X1k/view?usp=sharing

[Matthew Mosesohn (raytrac3r)]

9.0 ability to reproduce this is blocked by https://bugs.launchpad.net/fuel/+bug/1544505