URGENT - Running galera-install.yml playbook re-creating users and failing

[Satish Patel]

I am not sure other people having this issue but this is really URGENT issue for me. My production cluster is up and running and today i have tried to make some change in mysqlchk script and re-run playbook failing on "Creat galera user" and i found my all galera nodes are down.

Please let me know if this is a bug i don't mess with my galera in production or if i am doing something wrong please let me know.

TASK [galera_server : Create mysqlchk config] ***************************************************************************************************************************************
changed: [ostack-controller-03_galera_container-b1a3dceb]

TASK [galera_server : Add galera service check to services] *************************************************************************************************************************
ok: [ostack-controller-03_galera_container-b1a3dceb]

RUNNING HANDLER [galera_server : Restart xinetd] ************************************************************************************************************************************
changed: [ostack-controller-03_galera_container-b1a3dceb]

TASK [galera_server : include_tasks] ************************************************************************************************************************************************
included: /etc/ansible/roles/galera_server/tasks/galera_setup.yml for ostack-controller-03_galera_container-b1a3dceb

TASK [galera_server : Create galera users] ******************************************************************************************************************************************
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}

NO MORE HOSTS LEFT ******************************************************************************************************************************************************************

PLAY RECAP **************************************************************************************************************************************************************************
ostack-controller-01_galera_container-966ec0a6 : ok=5 changed=0 unreachable=0 failed=0
ostack-controller-02_galera_container-42b403be : ok=5 changed=0 unreachable=0 failed=0
ostack-controller-03_galera_container-b1a3dceb : ok=49 changed=8 unreachable=0 failed=1

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

[Satish Patel (satish-txt)]

I have noticed it did re-install mariadb again, is this normal ?? why this playbook doesn't detected mariadb was already installed

Installed date is today "Install Date: Wed 08 Aug 2018 12:15:46 PM EDT"

[root@ostack-controller-03-galera-container-b1a3dceb /]# rpm -qi MariaDB-server-10.1.30-1.el7.centos.x86_64
Name : MariaDB-server
Version : 10.1.30
Release : 1.el7.centos
Architecture: x86_64
Install Date: Wed 08 Aug 2018 12:15:46 PM EDT
Group : Applications/Databases
Size : 473695121
License : GPLv2
Signature : DSA/SHA1, Fri 22 Dec 2017 01:50:55 PM EST, Key ID cbcb082a1bb943db
Source RPM : MariaDB-server-10.1.30-1.el7.centos.src.rpm
Build Date : Thu 21 Dec 2017 12:27:17 PM EST
Build Host : centos70-x86-64
Relocations : (not relocatable)
Vendor : Monty Program AB
URL : http://mariadb.org
Summary : MariaDB: a very fast and robust SQL database server
Description :

Here is the full logs, showing re-installing stuff http://paste.openstack.org/show/727652/

[Satish Patel (satish-txt)]

I have capture full logs here incase someone need for more details: http://paste.openstack.org/show/727656/

[Satish Patel (satish-txt)]

More investigating on this issue found "TASK [galera_server : Remove conflicting distro packages]" un-installing all my RPM package and re-installing later thinking you don't have mariadb installed..

I am using CentOS 7.5

Now question is why it think its conflicting package?

[Satish Patel (satish-txt)]

I found related issue but not sure is its impacting my cluster https://bugs.launchpad.net/openstack-ansible/+bug/1742206

[Satish Patel (satish-txt)]

This is the solution

https://review.openstack.org/#/c/578844/1/vars/redhat-7.yml