URGENT - Running galera-install.yml playbook re-creating users and failing

Bug #1786070 reported by Satish Patel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
New
Undecided
Unassigned

Bug Description

I am not sure other people having this issue but this is really URGENT issue for me. My production cluster is up and running and today i have tried to make some change in mysqlchk script and re-run playbook failing on "Creat galera user" and i found my all galera nodes are down.

Please let me know if this is a bug i don't mess with my galera in production or if i am doing something wrong please let me know.

TASK [galera_server : Create mysqlchk config] ***************************************************************************************************************************************
changed: [ostack-controller-03_galera_container-b1a3dceb]

TASK [galera_server : Add galera service check to services] *************************************************************************************************************************
ok: [ostack-controller-03_galera_container-b1a3dceb]

RUNNING HANDLER [galera_server : Restart xinetd] ************************************************************************************************************************************
changed: [ostack-controller-03_galera_container-b1a3dceb]

TASK [galera_server : include_tasks] ************************************************************************************************************************************************
included: /etc/ansible/roles/galera_server/tasks/galera_setup.yml for ostack-controller-03_galera_container-b1a3dceb

TASK [galera_server : Create galera users] ******************************************************************************************************************************************
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}
FAILED - RETRYING: Create galera users (3 retries left).
FAILED - RETRYING: Create galera users (2 retries left).
FAILED - RETRYING: Create galera users (1 retries left).
failed: [ostack-controller-03_galera_container-b1a3dceb] (item=None) => {"attempts": 3, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false, "failed": true}

NO MORE HOSTS LEFT ******************************************************************************************************************************************************************

PLAY RECAP **************************************************************************************************************************************************************************
ostack-controller-01_galera_container-966ec0a6 : ok=5 changed=0 unreachable=0 failed=0
ostack-controller-02_galera_container-42b403be : ok=5 changed=0 unreachable=0 failed=0
ostack-controller-03_galera_container-b1a3dceb : ok=49 changed=8 unreachable=0 failed=1

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

Revision history for this message
Satish Patel (satish-txt) wrote :

I have noticed it did re-install mariadb again, is this normal ?? why this playbook doesn't detected mariadb was already installed

Installed date is today "Install Date: Wed 08 Aug 2018 12:15:46 PM EDT"

[root@ostack-controller-03-galera-container-b1a3dceb /]# rpm -qi MariaDB-server-10.1.30-1.el7.centos.x86_64
Name : MariaDB-server
Version : 10.1.30
Release : 1.el7.centos
Architecture: x86_64
Install Date: Wed 08 Aug 2018 12:15:46 PM EDT
Group : Applications/Databases
Size : 473695121
License : GPLv2
Signature : DSA/SHA1, Fri 22 Dec 2017 01:50:55 PM EST, Key ID cbcb082a1bb943db
Source RPM : MariaDB-server-10.1.30-1.el7.centos.src.rpm
Build Date : Thu 21 Dec 2017 12:27:17 PM EST
Build Host : centos70-x86-64
Relocations : (not relocatable)
Vendor : Monty Program AB
URL : http://mariadb.org
Summary : MariaDB: a very fast and robust SQL database server
Description :

Here is the full logs, showing re-installing stuff http://paste.openstack.org/show/727652/

Revision history for this message
Satish Patel (satish-txt) wrote :

I have capture full logs here incase someone need for more details: http://paste.openstack.org/show/727656/

Revision history for this message
Satish Patel (satish-txt) wrote :

More investigating on this issue found "TASK [galera_server : Remove conflicting distro packages]" un-installing all my RPM package and re-installing later thinking you don't have mariadb installed..

I am using CentOS 7.5

Now question is why it think its conflicting package?

Revision history for this message
Satish Patel (satish-txt) wrote :

I found related issue but not sure is its impacting my cluster https://bugs.launchpad.net/openstack-ansible/+bug/1742206

Revision history for this message
Satish Patel (satish-txt) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.