PKG systemctl status does not work on CentOS 7

[Fernando Ipar]

On the following environment:

[root@galera1 ~]# cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)

[root@galera1 ~]# rpm -qa|grep -i percona
percona-xtrabackup-2.3.5-1.el7.x86_64
percona-release-0.1-3.noarch
Percona-XtraDB-Cluster-shared-56-5.6.32-25.17.1.el7.x86_64
Percona-XtraDB-Cluster-galera-3-3.17-1.rhel7.x86_64
Percona-XtraDB-Cluster-client-56-5.6.32-25.17.1.el7.x86_64
percona-toolkit-2.2.19-1.noarch
Percona-XtraDB-Cluster-server-56-5.6.32-25.17.1.el7.x86_64

And with a functioning bootstrapped cluster:

[root@galera1 ~]# mysql -e 'show global status like "wsrep_cluster%"'
+--------------------------+--------------------------------------+
| Variable_name | Value |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id | 5 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | be051240-b1d8-11e6-8d20-6e9b5065bc35 |
| wsrep_cluster_status | Primary |
+--------------------------+--------------------------------------+

systemctl status reports the wrong status:

[root@galera1 ~]# systemctl status mysql
mysql.service - Percona XtraDB Cluster
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; disabled)
   Active: failed (Result: exit-code) since Thu 2016-11-24 00:08:56 UTC; 15min ago

Nov 24 00:08:56 galera1 systemd[1]: Starting Percona XtraDB Cluster...
Nov 24 00:08:56 galera1 mysql-systemd[3974]: WARNING: Another instance of mysqld running on 3881, exiting..
Nov 24 00:08:56 galera1 systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 24 00:08:56 galera1 systemd[1]: Failed to start Percona XtraDB Cluster.
Nov 24 00:08:56 galera1 systemd[1]: Unit mysql.service entered failed state.
Nov 24 00:10:12 galera1 systemd[1]: Stopped Percona XtraDB Cluster.
Nov 24 00:20:46 galera1 systemd[1]: Stopped Percona XtraDB Cluster.

Note the WARNING about another instance running. systemctl stop also does not work, and this behavior persists if I stop msyqld with 'mysqladmin shutdown' and then bootstrap PXC on the node again. The bootstrap procedure used is 'systemctl start <email address hidden>'

Here's MySQL's configuration:

[root@galera1 ~]# cat /etc/my.cnf
# Template my.cnf for PXC
# Edit to your requirements.

[mysqld]

log_bin

binlog_format = ROW
innodb_buffer_pool_size = 100M
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 20M
innodb_file_per_table = 1
datadir = /var/lib/mysql

wsrep_cluster_address = gcomm://10.0.1.37,10.0.1.38,10.0.1.39
wsrep_provider = /usr/lib64/galera3/libgalera_smm.so

wsrep_slave_threads = 8
wsrep_cluster_name = Cluster
wsrep_node_name = 10.0.1.37
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = sstuser:sstpass

innodb_locks_unsafe_for_binlog = 1
innodb_autoinc_lock_mode = 2
bind_address = 10.0.1.37

[mysqld_safe]
pid-file = /run/mysqld/mysql.pid
syslog

!includedir /etc/my.cnf.d

[root@galera1 ~]# ls /etc/my.cnf.d/
[root@galera1 ~]#

[Jericho Rivera (jericho-rivera)]

Can confirm, however, this might be an issue with systemd (I guess). Will set to confirmed for now to let developers decide on next steps (eg code fix or doc update)

[root@c7b ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

Bootstrap node1
[root@c7b ~]# systemctl start <email address hidden>
[root@c7b ~]# mysql -e "show status like 'wsrep_%'" | egrep "cluster_size|cluster_status"
wsrep_cluster_size 1
wsrep_cluster_status Primary

[root@c7b ~]# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Nov 25 04:19:11 c7b systemd[1]: Stopped Percona XtraDB Cluster.
[root@c7b ~]# systemctl stop mysql
[root@c7b ~]# systemctl start mysql
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
[root@c7b ~]# systemctl status mysql
● mysql.service - Percona XtraDB Cluster
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2016-11-25 04:25:26 UTC; 7s ago
  Process: 1694 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=1/FAILURE)

Nov 25 04:25:26 c7b systemd[1]: Starting Percona XtraDB Cluster...
Nov 25 04:25:26 c7b mysql-systemd[1694]: WARNING: Another instance of mysqld running on 1645, exiting..
Nov 25 04:25:26 c7b systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 25 04:25:26 c7b systemd[1]: Failed to start Percona XtraDB Cluster.
Nov 25 04:25:26 c7b systemd[1]: Unit mysql.service entered failed state.
Nov 25 04:25:26 c7b systemd[1]: mysql.service failed.

[root@c7b ~]# systemctl | grep mysql
● mysql.service loaded failed failed Percona XtraDB Cluster
● <email address hidden> loaded failed failed Percona XtraDB Cluster with config /etc/sysconfig/mysql.boostrap
  <email address hidden> loaded active running Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap
  system-mysql.slice loaded active active system-mysql.slice

Check status with:
[root@c7b ~]# systemctl status <email address hidden>
● <email address hidden> - Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap
   Loaded: loaded (/usr/lib/systemd/system/mysql@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-11-25 04:23:41 UTC; 2min 38s ago
  Process: 1301 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=0/SUCCESS)
  Process: 1253 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
 Main PID: 1300 (mysqld_safe)
   CGroup: /<email address hidden>
           ├─1300 /bin/sh /usr/bin/mysqld_safe --basedir=/usr --wsrep-new-cluster
           ├─1645 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep-provider=/usr/lib64/galera3/libgalera_smm.so --wsrep-new-cluster --pid-file=...
           └─1646 logger -t mysqld -p daemon.error

Nov 25 ...

[Fernando Ipar (fipar)]

Jericho, thanks for checking!
I realized today that this only happens after bootstrap, so it may be me being unused to systemd, but I will let the devs decide.

[EvgeniyPatlan (evgeniy-patlan)]

The message that the service has been boostrapped has been added. So if you bootstraped node and trying to run mysql there will be a message in messages log that indicates that the process was bootstrapped and how to deal with it.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1941