[library] Errors in mysqld_safe.log for controller for HA mode

Bug #1331488 reported by Anastasia Palkina on 2014-06-18
44
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Sergii Golovatiuk

Bug Description

"build_id": "2014-06-17_18-06-05",
"mirantis": "yes",
"build_number": "258",
"ostf_sha": "1740b5ce42ea1893f7d3e2c6cc59720bdb77c007",
"nailgun_sha": "057bb88abab1048322ed0ff48d632f8caf146e5a",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "ba9e19a3822d9c1dcda2f4046f2f5e3e6ac505dd",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "ff050d23d8a845cd097f7aa617285da0ab1894f6"

1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add 3 controllers+cinder, 1 compute
4. Start deployment. It was successful
5. But there are errors in mysqld_safe.log for controller:

[root@fuel ~]# less /var/log/docker-logs/remote/node-6.domain.tld/mysqld_safe.log | grep ERR
Jun 18 12:20:11 err: ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1
Jun 18 12:20:11 err: 140618 11:20:11 [ERROR] Aborting
Jun 18 12:20:17 err: ERROR: 1050 Table 'plugin' already exists
Jun 18 12:20:17 err: 140618 11:20:17 [ERROR] Aborting

Anastasia Palkina (apalkina) wrote :
Changed in fuel:
importance: Undecided → Medium
Changed in fuel:
status: New → Confirmed
Dmitry Borodaenko (angdraug) wrote :

Based on the logs, some service tries to update its DB schema before galera is done starting (11:21), although according to puppet logs, the first occurence of db_sync is 10 minutes later (11:32). The OpenStack packages to be installed before galera is started are cinder (11:18), swift (11:19), dashboard (11:20), neutron-l3 (11:20), glance (11:20).

Bogdan Dobrelya (bogdando) wrote :

Addressing this issue requires a major refactoring in puppet manifests, hence should be addressed in a separated blueprint

Bogdan Dobrelya (bogdando) wrote :

1) This bug has too many clones, raising to high
2) Once the related BP implemented, the solution should be backported for 5.0.x and 4.1.x as well

Changed in fuel:
importance: Medium → High
Bogdan Dobrelya (bogdando) wrote :

yes. It have been marked as related and should be closed once the related BP implemented

Bogdan Dobrelya (bogdando) wrote :
Changed in fuel:
status: Confirmed → Invalid
Dmitry Pyzhov (dpyzhov) wrote :

Feature request

no longer affects: fuel/4.1.x
no longer affects: fuel/5.0.x
Bogdan Dobrelya (bogdando) wrote :

 In Fuel 5.x we have implemented simultaneouse deployment for controllers in HA, and blocking mysqldump based replications are the root cause of the deployment issues we have to address in this blueprint as well.

Raising to blocker

Changed in fuel:
status: Invalid → Triaged
importance: High → Critical

There are 2 ways to fix this issue:
1. Deploy controllers in order
2. Use non blocking SST method, to allow several nodes to get mysql data from primary controller. MySQL Galera should use xtrabackup. More details can be obtained at http://galeracluster.com/documentation-webpages/nodeprovisioning.html
This will allow to have MySQL Galera running on Primary controller. Additional change that should be done is a good cluster check script for HAProxy which will route traffic to backup which is Primary Controller on Deployment stage. This 2 changes will allow to bootstrap/deploy controllers in parallel rather than consequently.

tags: added: ha to-be-covered-by-tests
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)
Dmitry Ilyin (idv1985) on 2014-07-15
summary: - Errors in mysqld_safe.log for controller for HA mode
+ [puppet] Errors in mysqld_safe.log for controller for HA mode
Dmitry Ilyin (idv1985) on 2014-07-15
summary: - [puppet] Errors in mysqld_safe.log for controller for HA mode
+ [library] Errors in mysqld_safe.log for controller for HA mode
Changed in fuel:
status: Triaged → In Progress
Changed in fuel:
status: In Progress → Fix Committed
Bogun Dmitriy (dbogun) wrote :

As for me, this bug is not related to any OpenStack component. The query, logged with "syntax error" alter mysql.user table. There is no OpenStack component that deal with this table directly.

This is the bug from ubuntu, with same error, with same query: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708176 I believe we need made some more research and find root cause of this issue.

PS Some more bugs with same query:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688792

Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Fuel DevOps (fuel-devops)
assignee: Fuel DevOps (fuel-devops) → Sergii Golovatiuk (sgolovatiuk)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.