Fail to create haproxy mysql user

[Mick Thompson]

I applied patch from 1491751.

TASK: [mariadb | Creating haproxy mysql user] *********************************
failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_ansible", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.169.107.0' login_port='3306' login_user='root' login_password='password' name='haproxy' password='' host='%'"], "delta": "0:00:00.330048", "end": "2015-09-03 11:59:14.579135", "failed": true, "rc": 2, "start": "2015-09-03 11:59:14.249087", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, \\\"Access denied for user 'root'@'mick-workstation' (using password: YES)\\\")\"", "}"], "warnings": []}
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, \"Access denied for user 'root'@'mick-workstation' (using password: YES)\")"
}
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

running with --step allows it to progress.

[Mick Thompson (michael-a-thompson)]

A further note: this occurs on my bare metal set-up. It does not occur on my VM set-up.

[Steven Dake (sdake)]

Mick,

Is the time on your bare metal machines synchronized and set properly? Could you provide more information, (centos + source? model or something else?), how many nodes you have in your bare metal setup, how many nodes you have in your virt setup, provide your inventory file as an attachment?

[Paul Bourke (pauldbourke)]

Multiple people currently reporting this in an AIO setup. Default centos+binary in a VM.

[Steven Dake (sdake)]

I suspect mysql isn't booted and the VM is setup with one processor, so a priority inversion is happening not permitting mysql to schedule enough to get to an operational state before ansible tries to create the haproxy database user.

Recommend try adding a wait for for port 3306 at the end of mysql (the next step after mysql orchestration is the creation of this database user).

http://docs.ansible.com/ansible/wait_for_module.html

[sean mooney (sean-k-mooney)]

i am also seeing this in a ubuntu vm with 2 vCpus AIO setup ubuntu base + source install

[sean mooney (sean-k-mooney)]

actully re reading my error message it is a differnet issue but may be related

failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_ansible", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.0.2.15' login_port='3306' login_user='root' login_password='password' name='haproxy' password='' host='%'"], "delta": "0:00:00.355437", "end": "2015-09-10 07:57:29.293101", "failed": true, "rc": 2, "start": "2015-09-10 07:57:28.937664", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \\\"Host '10.0.2.15' is not allowed to connect to this MariaDB server\\\")\"", "}"], "warnings": []}
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \"Host '10.0.2.15' is not allowed to connect to this MariaDB server\")"
}
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

[Ryan Hallisey (rthall14)]

AIO with master works again. One of the recent patches seems to have fixed this.

[Ryan Hallisey (rthall14)]

Report back if you have any further issues, but master AIO worked for me

[Sam Yaple (s8m)]

Let me start by saying this failure is _not_ relates to haproxy. The haproxy user is being created, but that is being done directly to the host outside of haproxy.

What is happening is mysql is not listening or responding on that box. The bootstrap process has failed or it is still syncing data from other boxes with galera. Without being on the box and with no additional information it is hard to tell what is causing the issue, however the message is fairly explicit "unable to connect to database". Since that isn't going through haproxy it truly is unable to connect to the database, i'd start there.

Given the inconsistency of the issue, this is likely a race condition or an improperly cleaned up environment (data container persisted between environment rekicks)

[Steven Dake (sdake)]

If anyone still experiences this issue, please report back on this thread, otherwise I'm going to close it out. We have made numerous synchronization fixes to the code base which may have resolved this problem, and nobody else seems to be experiencing the problem with current master.

Thanks
-steve

[Jeff Peeler (jpeeler-z)]

For what it's worth, I just reproduced this and dug into why it was breaking for me. Sam was correct about the database not being up and for me the container had failed and exited. The reason why was due to improper selinux labeling on /etc/kolla/mariadb/galera.cnf:

restorecon reset /etc/kolla/mariadb/galera.cnf context unconfined_u:object_r:user_home_t:s0->unconfined_u:object_r:etc_t:s0

Once fixed, I was able to proceed with my minimal deployment. Although this file is not the only one mislabeled, I think this can be closed if it's confirmed the reporter was running with selinux enforcing as well. Or alternatively, maybe we should consider supporting selinux (I assume we don't currently).

[Liyi Meng (liyi-meng)]

Hi Jeff, I run ubuntu which use apparmor, not selinux. I don't see any security enforcement there. But I still get the same error. Any suggestion on what to look in?

[Ashish (ashish-jain14)]

I have a AIO setup, I have seen this issue yesterday and here is how I got this working.
1) Login into mysql container and login into mysql prompt as root user
2) Run the command "select host from mysql.user where User = 'root';
This will return 3 values - localhost,127.0.0.1 & hostname of your machine
3) When I look into /etc/hosts of my host I did not had the mapping b/w my hostname and ip address of my host
4) Add mapping of host & ip into /etc/hosts
5) Clean up all the containers using the script kolla/tools/cleanup-containers
6) Restart the kolla deploy

Hope this helps.

[Salvatore Orlando (salvatore-orlando)]

I hit this bug too, both in the form mentioned in the report and in the one mentioned by Sean in comment #6.

In the first case the problem was that the host was running a Mysql server instance and therefore the mariadb container was not happy about it. Root case for this problem was PEBCAK. The problem was identified by simply running:

docker logs mariadb

Then the error in the 2nd form appeared.

failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_ansible", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.127.1.6' login_port='3306' login_user='root' login_password='password' name='haproxy' password='' host='%'"], "delta": "0:00:00.705690", "end": "2015-11-18 06:57:47.981298", "failed": true, "rc": 2, "start": "2015-11-18 06:57:47.275608", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \\\"Host 'host-10-127-1-6.openstacklocal' is not allowed to connect to this MariaDB server\\\")\"", "}"], "warnings": []}
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \"Host 'host-10-127-1-6.openstacklocal' is not allowed to connect to this MariaDB server\")"
}

However, right before that I noticed that:

TASK: [mariadb | Sleeping for 15 seconds while the cluster starts] ************
skipping: [localhost]

This step should have probably not been skipped since the cluster needed actually to be started.
Anyway I worked around the issue in this way:

docker stop mariadb mariadb_data
docker rm mariadb mariadb_data
sudo kolla-ansible deploy

and was able to make progress until the next error ;)

[Sidharth Surana (ssurana)]

I hit this bug as well, when trying out the minimal install for all-in-one inventory

===============
TASK: [mariadb | Cleaning up temp file on localhost] **************************
ok: [localhost -> 127.0.0.1]

TASK: [mariadb | Starting MariaDB data container] *****************************
changed: [localhost]

TASK: [mariadb | Starting MariaDB bootstrap container] ************************
changed: [localhost]

TASK: [mariadb | Sleeping for 15 seconds while the cluster starts] ************
ok: [localhost]

TASK: [mariadb | Starting MariaDB container] **********************************
changed: [localhost]

TASK: [mariadb | Creating haproxy mysql user] *********************************
failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_ansible", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.161.16.77' login_port='3306' login_user='root' login_password='password' name='haproxy' password='' host='%'"], "delta": "0:00:00.718363", "end": "2015-12-08 05:15:05.505876", "failed": true, "rc": 2, "start": "2015-12-08 05:15:04.787513", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, \\\"Access denied for user 'root'@'sc-a01-050-222' (using password: YES)\\\")\"", "}"], "warnings": []}
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, \"Access denied for user 'root'@'sc-a01-050-222' (using password: YES)\")"
}
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/site.retry

localhost : ok=51 changed=3 unreachable=0 failed=1

Command failed ansible-playbook -i /ssurana/kolla/tools/../ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml /ssurana/kolla/tools/../ansible/site.yml
==========================

My setup is source based, ubuntu trusty install from the master branch.

Further debugging suggests that there is potentially some issue in the bootstrapping step.
Seems like the "mysql_secure_installation" step is not getting through properly.

See logs from the container below:
===================================================================
root:~# docker logs -f mariadb
INFO:__main__:Kolla config strategy set to: COPY_ONCE
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /etc/mysql/my.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/galera.cnf to /etc/mysql/my.cnf
INFO:__main__:Setting permissions for /etc/mysql/my.cnf
INFO:__main__:Writing out command to execute
Installing MariaDB/MySQL system tables in ...

[Sidharth Surana (ssurana)]

Fix proposed https://review.openstack.org/#/c/254961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254961
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=37d44444d7bc1804976fc4477c4022ee6cc06e44
Submitter: Jenkins
Branch: master

commit 37d44444d7bc1804976fc4477c4022ee6cc06e44
Author: Sidharth Surana <email address hidden>
Date: Tue Dec 8 02:23:58 2015 -0800

    Make galeradb bootstraping robust

    Currently, there are arbitrary wait for mariadb service startup.
    However, this leads to nondeterministic results in the current
    workflow. This patch tries to make the workflow more deterministic.

    Change-Id: I3c6245cce93c7ff0d3d57cb2ae065a1ed1487769
    Closes-Bug: #1491782

[Vikram Hosakote (vhosakot)]

I saw this error too today on Ubuntu 14.04 with the latest kolla Mitaka code.

TASK: [mariadb | Creating haproxy mysql user] *********************************
failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='192.168.122.201' login_port='3306' login_user='root' login_password='password' name='haproxy' password='' host='%'"], "delta": "0:00:00.767763", "end": "2016-03-01 21:35:40.843791", "failed": true, "rc": 2, "start": "2016-03-01 21:35:40.076028", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \\\"Host '192.168.122.201' is not allowed to connect to this MariaDB server\\\")\"", "}"], "warnings": []}
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1130, \"Host '192.168.122.201' is not allowed to connect to this MariaDB server\")"
}
msg: Task failed as maximum retries was encountered

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/home/kolla/site.retry

localhost : ok=69 changed=1 unreachable=0 failed=1

Command failed ansible-playbook -i /usr/local/share/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e action=deploy /usr/local/share/kolla/ansible/site.yml --verbose --verbose --verbose --verbose
kolla@kolla-ubuntu:~$

================

I was able to resolve this issue after following the steps given by Ashish (ashish-jain14) above in comment #13.

kolla@kolla-ubuntu:~$ sudo docker exec -i -t mariadb mysql -u root -ppassword -h 192.168.122.201

MariaDB [(none)]> use mysql;

MariaDB [mysql]> select host from mysql.user where User = 'root';
+--------------+
| host |
+--------------+
| 127.0.0.1 |
| ::1 |
| kolla-ubuntu |
| localhost |
+--------------+
4 rows in set (0.01 sec)

MariaDB [mysql]> exit

kolla-ubuntu is my hostname and 192.168.122.201 is my docker registry's IP address.

Now, replace "127.0.1.1 kolla-ubuntu" with "192.168.122.201 kolla-ubuntu" in /etc/hosts and save it.

docker rm -f mariadb

sudo kolla-ansible deploy

[Swapnil Kulkarni (coolsvap-deactivatedaccount)]

I am still facing this issue after applying workaround in comment #18

[Steven Dake (sdake)]

comment #18 had selinux enforcing and VIP == host ip which is PEBKAC.

[Steven Dake (sdake)]

the root cause here is caused by 127.0.1.1 having the machine name in it. This is standard installation practice on ubuntu. instead the address should be the machine's actual IP address in the /etc/hosts file. The solution to this is to document that the build node nd deployment targets should all have matching /etc/hosts file and they should contain all hosts in the network related to the deployment.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Swapnil Kulkarni (coolsvap) (<email address hidden>) on branch: master
Review: https://review.openstack.org/300955

[Michał Jastrzębski (inc007)]

Is this bug still apearing? Can someone reproduce or confirm it?

[Clayton Gonsalves (clayg)]

Came across this bug in AIO CentOS setup.

TASK [mariadb : Creating haproxy mysql user] ***********************************
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (9 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (8 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (7 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (6 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (5 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (4 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (3 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (2 retries left).
FAILED - RETRYING: TASK: mariadb : Creating haproxy mysql user (1 retries left).
fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.0.3.15' login_port='3306' login_user='root' login_password='lbKnSiQT3xOWbsvrDuPdvnJuHGmTMTCXun2SHhc9' name='haproxy' password='' host='%' priv=*.*:USAGE"], "delta": "0:00:01.311719", "end": "2016-07-15 01:13:55.112196", "failed": true, "rc": 2, "start": "2016-07-15 01:13:53.800477", "stderr": "", "stdout": "localhost | FAILED! => {\r\n \"changed\": false, \r\n \"failed\": true, \r\n \"msg\": \"unable to connect to database, check login_user and login_password are correct or /home/ansible/.my.cnf has the credentials. Exception message: (1045, \\\"Access denied for user 'root'@'localhost.localdomain' (using password: YES)\\\")\"\r\n}", "stdout_lines": ["localhost | FAILED! => {", " \"changed\": false, ", " \"failed\": true, ", " \"msg\": \"unable to connect to database, check login_user and login_password are correct or /home/ansible/.my.cnf has the credentials. Exception message: (1045, \\\"Access denied for user 'root'@'localhost.localdomain' (using password: YES)\\\")\"", "}"], "warnings": []}
 to retry, use: --limit @/usr/share/kolla/ansible/site.retry

PLAY RECAP *********************************************************************
localhost : ok=55 changed=9 unreachable=0 failed=1

Command failed ansible-playbook -i /usr/share/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla/ansible/site.yml

[Swapnil Kulkarni (coolsvap-deactivatedaccount)]

Clayton, can you please apply steps in https://bugs.launchpad.net/kolla/+bug/1491782/comments/21 and confirm if it works for you.

Thanks

[Yee-Ting Li (yee379)]

Hi, i get the same error with kolla 1.1.2 with builds with centos / binary.

attempting to log onto the mariadb manually as per vhosakot in #18, also fails. i'm using the password defined in /etc/kolla/password.yml / database_password (which is the same password that kolla-ansible is attempting to use):

# sudo docker exec -i -t mariadb /bin/bash
(mariadb)[mysql@kolla-test /]$ mysql -u root -p -h <a>.<b>.<c>.<d>
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'<DNS>' (using password: YES)