[library] Errors in mysqld_safe.log for controller for HA mode

[Anastasia Palkina]

"build_id": "2014-06-17_18-06-05",
"mirantis": "yes",
"build_number": "258",
"ostf_sha": "1740b5ce42ea1893f7d3e2c6cc59720bdb77c007",
"nailgun_sha": "057bb88abab1048322ed0ff48d632f8caf146e5a",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "ba9e19a3822d9c1dcda2f4046f2f5e3e6ac505dd",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "ff050d23d8a845cd097f7aa617285da0ab1894f6"

1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add 3 controllers+cinder, 1 compute
4. Start deployment. It was successful
5. But there are errors in mysqld_safe.log for controller:

[root@fuel ~]# less /var/log/docker-logs/remote/node-6.domain.tld/mysqld_safe.log | grep ERR
Jun 18 12:20:11 err: ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1
Jun 18 12:20:11 err: 140618 11:20:11 [ERROR] Aborting
Jun 18 12:20:17 err: ERROR: 1050 Table 'plugin' already exists
Jun 18 12:20:17 err: 140618 11:20:17 [ERROR] Aborting

[Dmitry Borodaenko (angdraug)]

Based on the logs, some service tries to update its DB schema before galera is done starting (11:21), although according to puppet logs, the first occurence of db_sync is 10 minutes later (11:32). The OpenStack packages to be installed before galera is started are cinder (11:18), swift (11:19), dashboard (11:20), neutron-l3 (11:20), glance (11:20).

[Bogdan Dobrelya (bogdando)]

Addressing this issue requires a major refactoring in puppet manifests, hence should be addressed in a separated blueprint

[Bogdan Dobrelya (bogdando)]

1) This bug has too many clones, raising to high
2) Once the related BP implemented, the solution should be backported for 5.0.x and 4.1.x as well

[Dmitry Borodaenko (angdraug)]

Is this bug superceded by https://blueprints.launchpad.net/fuel/+spec/deployment-stage-galera?

[Bogdan Dobrelya (bogdando)]

yes. It have been marked as related and should be closed once the related BP implemented

[Bogdan Dobrelya (bogdando)]

Superceeded by BP https://blueprints.launchpad.net/fuel/+spec/deployment-stage-galera

[Dmitry Pyzhov (dpyzhov)]

Feature request

[Bogdan Dobrelya (bogdando)]

 In Fuel 5.x we have implemented simultaneouse deployment for controllers in HA, and blocking mysqldump based replications are the root cause of the deployment issues we have to address in this blueprint as well.

Raising to blocker

[Bogdan Dobrelya (bogdando)]

No more superceeded by BP https://blueprints.launchpad.net/fuel/+spec/deployment-stage-galera

[Sergii Golovatiuk (sgolovatiuk)]

There are 2 ways to fix this issue:
1. Deploy controllers in order
2. Use non blocking SST method, to allow several nodes to get mysql data from primary controller. MySQL Galera should use xtrabackup. More details can be obtained at http://galeracluster.com/documentation-webpages/nodeprovisioning.html
This will allow to have MySQL Galera running on Primary controller. Additional change that should be done is a good cluster check script for HAProxy which will route traffic to backup which is Primary Controller on Deployment stage. This 2 changes will allow to bootstrap/deploy controllers in parallel rather than consequently.

[Bogun Dmitriy (dbogun)]

As for me, this bug is not related to any OpenStack component. The query, logged with "syntax error" alter mysql.user table. There is no OpenStack component that deal with this table directly.

This is the bug from ubuntu, with same error, with same query: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708176 I believe we need made some more research and find root cause of this issue.

PS Some more bugs with same query:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688792