Galera, Mariadb 10.1 and systemd don't work anymore

Bug #1747475 reported by Gaëtan Trellu
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
In Progress
Undecided
Gaëtan Trellu

Bug Description

The way how Trove is spawning Galera cluster with MariaDB 10.1 and systemd is not working anymore.
In MariaDB 10.1 the <email address hidden>[1] systemd unit file is deprecated.

MariaDB recommends to use "galera_new_cluster"[2] command to bootstrap a Galera cluster.

/var/log/trove/guestagent.log
(trove.guestagent.datastore.galera_common.service): 2018-02-05 17:22:32,519 DEBUG bootstrap the instance? : True
(trove.guestagent.datastore.galera_common.service): 2018-02-05 17:22:32,520 INFO Bootstraping cluster.
(oslo_concurrency.processutils): 2018-02-05 17:22:32,521 DEBUG Running cmd (subprocess): sudo systemctl start <email address hidden>
(oslo_concurrency.processutils): 2018-02-05 17:22:32,646 DEBUG CMD "sudo systemctl start <email address hidden>" returned: 1 in 0.125s
(oslo_concurrency.processutils): 2018-02-05 17:22:32,647 DEBUG u'sudo systemctl start <email address hidden>' failed. Not Retrying.
(trove.guestagent.datastore.galera_common.manager): 2018-02-05 17:22:32,648 ERROR Cluster installation failed.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/galera_common/manager.py", line 55, in install_cluster
    replication_user, cluster_configuration, bootstrap)
  File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/galera_common/service.py", line 72, in install_cluster
    self._bootstrap_cluster(timeout=CONF.restore_usage_timeout)
  File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/galera_common/service.py", line 53, in _bootstrap_cluster
    shell=True, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/trove/common/utils.py", line 237, in execute_with_timeout
    return execute(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
    cmd=sanitized_cmd)
ProcessExecutionError: Unexpected error while running command.
Command: sudo systemctl start <email address hidden>
Exit code: 1
Stdout: u''
Stderr: u'Job for <email address hidden> failed because the control process exited with error code. See "systemctl status <email address hidden>" and "journalctl -xe" for details.\n'
(trove.guestagent.datastore.service): 2018-02-05 17:22:32,652 DEBUG Prepare has not completed yet, skipping heartbeat.

# systemctl status mariadb\@bootstrap.service
● <email address hidden> - MariaDB 10.1.30 database server (multi-instance)
   Loaded: loaded (/lib/systemd/system/mariadb@.service; disabled; vendor preset: enabled)
  Drop-In: /<email address hidden>
           └─use_galera_new_cluster.conf
   Active: failed (Result: exit-code) since Mon 2018-02-05 17:22:32 UTC; 3min 13s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 1955 ExecStart=/usr/bin/false (code=exited, status=1/FAILURE)
  Process: 1951 ExecStart=/usr/bin/echo Please use galera_new_cluster to start the mariadb service with --wsrep-new-cluster (code=exited, status=0/SUCCESS)
  Process: 1938 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery --defaults-file=/etc/mysql/conf.d/my%I.cnf`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION%I=$VAR ||
  Process: 1932 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION%I (code=exited, status=0/SUCCESS)
 Main PID: 1955 (code=exited, status=1/FAILURE)

Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: Starting MariaDB 10.1.30 database server (multi-instance)...
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: Could not open required defaults file: /etc/mysql/conf.d/mybootstrap.cnf
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: Fatal error in defaults handling. Program aborted
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: WARNING: Defaults file '/etc/mysql/conf.d/mybootstrap.cnf' not found!
Feb 05 17:22:32 mariadb-member-1517851079 echo[1951]: Please use galera_new_cluster to start the mariadb service with --wsrep-new-cluster
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: Failed to start MariaDB 10.1.30 database server (multi-instance).
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Unit entered failed state.
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
...skipping...
● <email address hidden> - MariaDB 10.1.30 database server (multi-instance)
   Loaded: loaded (/lib/systemd/system/mariadb@.service; disabled; vendor preset: enabled)
  Drop-In: /<email address hidden>
           └─use_galera_new_cluster.conf
   Active: failed (Result: exit-code) since Mon 2018-02-05 17:22:32 UTC; 3min 13s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 1955 ExecStart=/usr/bin/false (code=exited, status=1/FAILURE)
  Process: 1951 ExecStart=/usr/bin/echo Please use galera_new_cluster to start the mariadb service with --wsrep-new-cluster (code=exited, status=0/SUCCESS)
  Process: 1938 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery --defaults-file=/etc/mysql/conf.d/my%I.cnf`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION%I=$VAR ||
  Process: 1932 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION%I (code=exited, status=0/SUCCESS)
 Main PID: 1955 (code=exited, status=1/FAILURE)

Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: Starting MariaDB 10.1.30 database server (multi-instance)...
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: Could not open required defaults file: /etc/mysql/conf.d/mybootstrap.cnf
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: Fatal error in defaults handling. Program aborted
Feb 05 17:22:32 mariadb-member-1517851079 sh[1938]: WARNING: Defaults file '/etc/mysql/conf.d/mybootstrap.cnf' not found!
Feb 05 17:22:32 mariadb-member-1517851079 echo[1951]: Please use galera_new_cluster to start the mariadb service with --wsrep-new-cluster
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: Failed to start MariaDB 10.1.30 database server (multi-instance).
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Unit entered failed state.
Feb 05 17:22:32 mariadb-member-1517851079 systemd[1]: <email address hidden>: Failed with result 'exit-code'.

[1]https://github.com/MariaDB/server/blob/10.1/support-files/use_galera_new_cluster.conf
[2]https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/#bootstrapping-a-new-cluster

Changed in trove:
assignee: nobody → Gaëtan Trellu (goldyfruit)
Revision history for this message
Pierre Blanc (pierreblanc75) wrote :

I don't think zuul test the clustering.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/541382

Changed in trove:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.