ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) errors during cluster deployment.

[Alexey Galkin]

Deployment fails with message:
"Deployment has failed. All nodes are finished. Failed tasks: Task[database/2] Stopping the deployment process!"

Here is what I've found in the puppet logs

2016-04-18 16:21:18 ERR Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
2016-04-18 16:21:18 ERR (/Stage[main]/Openstack::Galera::Status/Mysql_user[clustercheck@192.168.0.3]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'clustercheck'@'192.168.0.3' IDENTIFIED BY PASSWORD '*4A84AA3DE1A5BD3479D0E4A5F99F1C213F29F019'' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
2016-04-18 16:21:18 ERR (/Stage[main]/Openstack::Galera::Status/Mysql_user[clustercheck@192.168.0.3]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'clustercheck'@'192.168.0.3' IDENTIFIED BY PASSWORD '*4A84AA3DE1A5BD3479D0E4A5F99F1C213F29F019'' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Steps To Reproduce:

1. Install fuel
2. Deploy enviroment

Actual result:

Deploy failed with the errors:

2016-04-18 16:21:18 ERR Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
2016-04-18 16:21:18 ERR (/Stage[main]/Openstack::Galera::Status/Mysql_user[clustercheck@192.168.0.3]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'clustercheck'@'192.168.0.3' IDENTIFIED BY PASSWORD '*4A84AA3DE1A5BD3479D0E4A5F99F1C213F29F019'' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
2016-04-18 16:21:18 ERR (/Stage[main]/Openstack::Galera::Status/Mysql_user[clustercheck@192.168.0.3]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'clustercheck'@'192.168.0.3' IDENTIFIED BY PASSWORD '*4A84AA3DE1A5BD3479D0E4A5F99F1C213F29F019'' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Reproducibility:

Floating bug

Description of the environment:

 Shotgun2 report: http://paste.openstack.org/show/494463/
 Operation system: Ubuntu
 Network: Neutron+Vlan
 Fuel: 9.0-216

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Timur Nurlygayanov (tnurlygayanov)]

Related issue:
https://bugs.launchpad.net/fuel/+bug/1548271
https://bugs.launchpad.net/fuel/+bug/1569990

[Anastasia Kuznetsova (akuznetsova)]

Issue was reproduced again on ISO 254 on the following scenario:
            1. Create cluster. Set install Ceilometer option
            2. Add 3 node with controller role
            3. Add 1 nodes with compute role
            4. Add 1 node with mongo role
            5. Deploy the cluster

Deployment has failed with:
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[database/3], Task[database/5] Stopping the deployment process!'

Following errors were found on the one of nodes:
2016-04-27 22:02:14 +0000 Puppet (err): Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2016-04-27 22:02:14 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'clustercheck'@'10.109.1.7' IDENTIFIED BY PASSWORD '*CDB24FDB36375338E104A5962C319A5CE9B4C721'' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

[Vladyslav Drok (vdrok)]

More failures here - https://bugs.launchpad.net/fuel/+bug/1577893

[Timur Nurlygayanov (tnurlygayanov)]

Any updates?

[Sergii Golovatiuk (sgolovatiuk)]

The fix is being tested.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313058

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313058
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6d9fd6c5b6fab8f3b174fc2fe04e708cd54c8a2f
Submitter: Jenkins
Branch: master

commit 6d9fd6c5b6fab8f3b174fc2fe04e708cd54c8a2f
Author: Sergii Golovatiuk <email address hidden>
Date: Thu May 5 18:44:52 2016 +0200

    Create clustercheck after mysql-server

    - Under some circumstances clustercheck user/files are created before
      mysql-server is fully set up. This patch changes the behavior to
      create user after mysql-server
    - Create my.cnf primary controller. It only helps to bootstrap the
      cluster. After all we link it to our my.localhost.cnf. We don't need
      to set it up on non primary controllers.

    Change-Id: I97de2a01b79851eeca34524c92f406c99ea56cda
    Closes-Bug: 1571763
    Closes-Bug: 1577462
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314038
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c5d7e98587297c3291a5bdefe6f1f8a326784079
Submitter: Jenkins
Branch: stable/mitaka

commit c5d7e98587297c3291a5bdefe6f1f8a326784079
Author: Sergii Golovatiuk <email address hidden>
Date: Thu May 5 18:44:52 2016 +0200

    Create clustercheck after mysql-server

    - Under some circumstances clustercheck user/files are created before
      mysql-server is fully set up. This patch changes the behavior to
      create user after mysql-server
    - Create my.cnf primary controller. It only helps to bootstrap the
      cluster. After all we link it to our my.localhost.cnf. We don't need
      to set it up on non primary controllers.

    Change-Id: I97de2a01b79851eeca34524c92f406c99ea56cda
    Closes-Bug: 1571763
    Closes-Bug: 1577462
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[Dmitriy Kruglov (dkruglov)]

The issue was verified on runs of the corresponding swarm tests. No failures on 10 runs in a row.