[system_tests] Deploy ha flat on ubuntu fails on sync galera on controllers

[Tatyanka]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "389"
  build_id: "2014-08-03_02-01-14"
  astute_sha: "ce86172e77661026c91fdf1ff8066d7df1f7d89d"
  fuellib_sha: "4e3fdd75f8dabde8e5d07067545d8043a70a176b"
  ostf_sha: "a3fa823ea0e4e03beb637ae07a91adea82c33182"
  nailgun_sha: "bd0127be0061029f9f910547db5e633c82244942"
  fuelmain_sha: "e99879292cf6e96b8991300d947df76b69134bb1"

Cluster in error state with error in puppet on galera exec wait sync
http://paste.openstack.org/show/90526/

[Sergii Golovatiuk (sgolovatiuk)]

According to logs,

MySQL with Galera was not able to Initiate transfer on node-2

Here is the log from node-1
2014-08-03T17:11:17.120625+01:00 info: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
2014-08-03T17:11:25.101678+01:00 info: INFO: MySQL started
2014-08-03T17:11:25.245771+01:00 info: INFO: GTID OK: ef86fc0c-1b26-11e4-8e83-eb36c97e0a94:968
2014-08-03T17:12:25.302240+01:00 info: INFO: GTID OK: ef86fc0c-1b26-11e4-8e83-eb36c97e0a94:998
2014-08-03T17:13:25.363481+01:00 info: INFO: GTID OK: ef86fc0c-1b26-11e4-8e83-eb36c97e0a94:1028
2014-08-03T17:14:27.217568+01:00 info: INFO: GTID OK: ef86fc0c-1b26-11e4-8e83-eb36c97e0a94:1059

It seems to be good

Logs from node-4

2014-08-03T17:25:22.330226+01:00 info: INFO: MySQL started
2014-08-03T17:25:22.448313+01:00 err: ERROR: GTID have wrong format: wsrep_local_state_uuid:18446744073709551615
2014-08-03T17:25:22.449267+01:00 err: ERROR: Wrong GTID, not updating gtid attribute
2014-08-03T17:27:09.164154+01:00 err: ERROR: GTID have wrong format: wsrep_local_state_uuid:18446744073709551615
2014-08-03T17:27:09.164154+01:00 err: ERROR: Wrong GTID, not updating gtid attribute
2014-08-03T17:27:23.411150+01:00 info: INFO: GTID OK: ef86fc0c-1b26-11e4-8e83-eb36c97e0a94:1430

Logs from node-2
014-08-03T17:25:14.335806+01:00 info: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
2014-08-03T17:25:19.629571+01:00 info: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 1 retries left
2014-08-03T17:25:27.391246+01:00 info: INFO: MySQL started
2014-08-03T17:25:32.156082+01:00 err: ERROR: GTID have wrong format: wsrep_local_state_uuid:18446744073709551615
2014-08-03T17:25:32.157074+01:00 err: ERROR: Wrong GTID, not updating gtid attribute
2014-08-03T17:26:32.187558+01:00 err: ERROR: GTID have wrong format: wsrep_local_state_uuid:18446744073709551615
2014-08-03T17:26:32.198079+01:00 err: ERROR: Wrong GTID, not updating gtid attribute
2014-08-03T17:28:48.345912+01:00 err: ERROR: GTID have wrong format: :
2014-08-03T17:28:48.345912+01:00 err: ERROR: Wrong GTID, not updating gtid attribute
2014-08-03T17:28:48.345912+01:00 info: INFO: MySQL failed to stop after 15s using SIGTERM. Trying SIGKILL...
2014-08-03T17:28:48.345912+01:00 info: INFO: MySQL stopped

Since then there were attempts to transfer data but all of them were not successful.

I deleted everything from /var/log/mysql/*, generated a new structure and tried to start mysqld manually. It was not successful. It looks like it's a sporadic issue on Ubuntu and could be related to MySQL or galera packages themselves.

[Bogdan Dobrelya (bogdando)]

Moving to 6.0 cuz lowered from high to medium

[Tatyanka (tatyana-leontovich)]

reproduced on
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "424"
  build_id: "2014-08-12_02-01-17"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "54d834afccf1f8a97e4a82d7d081e7d1a1e068a1"
  ostf_sha: "d2a894d228c1f3c22595a77f04b1e00d09d8e463"
  nailgun_sha: "c7e00e5a00499d9f8dec608541dad1d745a8dd2e"
  fuelmain_sha: "9d4463400b4924159c978af43855e48bcf2a84b2"