When moving to containers we broke password changing

[Michele Baldessari]

Starting with queens (aka the move to containers) we broke changing at least mysqlrootpassword and the rabbitpassword. So as soon as an operator changes them the deployment will fail with messages like:
Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user'
returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

First seen in: https://bugzilla.redhat.com/show_bug.cgi?id=1609022

A lot of folks have been hit by this because of https://bugs.launchpad.net/tripleo/+bug/1790580 (i.e. using the derived param workflow would trigger a bunch of password changes)

[Alex Schultz (alex-schultz)]

It's likely that we could solve this by adding a task in Step 1 on stack update to sync the root password with the expected one. This is likely because we have lost the dependency that the password update action (in puppet) is performed prior to the updating of /root/.my.cnf. We'll need to replicate that logic itself in THT now

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/602499

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/602969

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/603612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603612
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8fe26972a0777f8e7bc7aa0e5e16774d1df5d15d
Submitter: Zuul
Branch: master

commit 8fe26972a0777f8e7bc7aa0e5e16774d1df5d15d
Author: Damien Ciabrini <email address hidden>
Date: Wed Sep 19 04:38:33 2018 -0400

    mysql: use clustercheck credentials to poll galera state

    In the galera container, clustercheck is currently configured to
    use root credentials to poll the state of the local galera node.

    configure clustercheck to use the clustercheck credentials instead.

    Change-Id: Icca33ec759a9ef5abca6da4cd2e59f0a5d9b7061
    Related-Bug: #1792416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602969
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8e67ec833173920ac60b5548a711885a4d28e16f
Submitter: Zuul
Branch: master

commit 8e67ec833173920ac60b5548a711885a4d28e16f
Author: Damien Ciabrini <email address hidden>
Date: Sun Sep 16 07:38:35 2018 -0400

    mysql: do not overwrite password file during docker-puppet

    During a stack update, when docker-puppet regenerates configs files for the
    mysql service, the root mysql passwords may change. Mysql has to update its
    internal state (e.g. password in mysql DB) to reflect the change, but this
    only happens when paunch restarts mysql; and the old password it required
    to until the change is applied.

    For such services, update the config hash to notify paunch that a restart is
    needed, but do not update the password file in docker-puppet and let the
    service's containers regenerate it instead.

    Change-Id: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Partial-Bug: #1792416

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/602499
Reason: Clearing the gate. Do not retore this until being given the all clear. See http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000368.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602499
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=467c6879d62b27225e3cb57a237491cd3505d92b
Submitter: Zuul
Branch: master

commit 467c6879d62b27225e3cb57a237491cd3505d92b
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 13 19:06:43 2018 -0400

    mysql: fix root password update for containerized mysql

    Since the mysql service has been containerized, we lost the ability
    to update the root password during a stack update.

    When the mysql root password in hiera differs from the one currently
    set in the mysql DB, connect to the DB with password from .my.cnf and
    update credentials of the root user before the puppet mysql module
    tries to access the database. Also update other root DB users.

    Change-Id: I8fe9a640ba36288a1f9cb18563b363159d4731c0
    Depends-On: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Closes-Bug: #1792416

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/635853

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/635883

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/635886

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635883
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9a59f1b0a23441d3e71671455b3fbef4d70f4ec4
Submitter: Zuul
Branch: stable/rocky

commit 9a59f1b0a23441d3e71671455b3fbef4d70f4ec4
Author: Damien Ciabrini <email address hidden>
Date: Sun Sep 16 07:38:35 2018 -0400

    mysql: do not overwrite password file during docker-puppet

    During a stack update, when docker-puppet regenerates configs files for the
    mysql service, the root mysql passwords may change. Mysql has to update its
    internal state (e.g. password in mysql DB) to reflect the change, but this
    only happens when paunch restarts mysql; and the old password it required
    to until the change is applied.

    For such services, update the config hash to notify paunch that a restart is
    needed, but do not update the password file in docker-puppet and let the
    service's containers regenerate it instead.

    Change-Id: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Partial-Bug: #1792416
    (cherry picked from commit 8e67ec833173920ac60b5548a711885a4d28e16f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635886
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ed2c3d22968901cb8d4a577613243e0450a15ff9
Submitter: Zuul
Branch: stable/rocky

commit ed2c3d22968901cb8d4a577613243e0450a15ff9
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 13 19:06:43 2018 -0400

    mysql: fix root password update for containerized mysql

    Since the mysql service has been containerized, we lost the ability
    to update the root password during a stack update.

    When the mysql root password in hiera differs from the one currently
    set in the mysql DB, connect to the DB with password from .my.cnf and
    update credentials of the root user before the puppet mysql module
    tries to access the database. Also update other root DB users.

    Change-Id: I8fe9a640ba36288a1f9cb18563b363159d4731c0
    Depends-On: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Closes-Bug: #1792416
    (cherry picked from commit 467c6879d62b27225e3cb57a237491cd3505d92b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635853
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b35b807670f68c757ed561c19b3467d778096129
Submitter: Zuul
Branch: stable/rocky

commit b35b807670f68c757ed561c19b3467d778096129
Author: Damien Ciabrini <email address hidden>
Date: Wed Sep 19 04:38:33 2018 -0400

    mysql: use clustercheck credentials to poll galera state

    In the galera container, clustercheck is currently configured to
    use root credentials to poll the state of the local galera node.

    configure clustercheck to use the clustercheck credentials instead.

    Change-Id: Icca33ec759a9ef5abca6da4cd2e59f0a5d9b7061
    Related-Bug: #1792416
    (cherry picked from commit 8fe26972a0777f8e7bc7aa0e5e16774d1df5d15d)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/637525

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637539

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637577

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637525
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=21876cec33e7a4af805f0005d7e23a0d9b7bc600
Submitter: Zuul
Branch: stable/queens

commit 21876cec33e7a4af805f0005d7e23a0d9b7bc600
Author: Damien Ciabrini <email address hidden>
Date: Wed Sep 19 04:38:33 2018 -0400

    mysql: use clustercheck credentials to poll galera state

    In the galera container, clustercheck is currently configured to
    use root credentials to poll the state of the local galera node.

    configure clustercheck to use the clustercheck credentials instead.

    Change-Id: Icca33ec759a9ef5abca6da4cd2e59f0a5d9b7061
    Related-Bug: #1792416
    (cherry picked from commit 8fe26972a0777f8e7bc7aa0e5e16774d1df5d15d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637539
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c9b06deaa1c25fdd663e5ce4792c28374d9b2eff
Submitter: Zuul
Branch: stable/queens

commit c9b06deaa1c25fdd663e5ce4792c28374d9b2eff
Author: Damien Ciabrini <email address hidden>
Date: Sun Sep 16 07:38:35 2018 -0400

    mysql: do not overwrite password file during docker-puppet

    During a stack update, when docker-puppet regenerates configs files for the
    mysql service, the root mysql passwords may change. Mysql has to update its
    internal state (e.g. password in mysql DB) to reflect the change, but this
    only happens when paunch restarts mysql; and the old password it required
    to until the change is applied.

    For such services, update the config hash to notify paunch that a restart is
    needed, but do not update the password file in docker-puppet and let the
    service's containers regenerate it instead.

    Change-Id: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Partial-Bug: #1792416
    (cherry picked from commit 8e67ec833173920ac60b5548a711885a4d28e16f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637577
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cc4f0a8fddc1fd8fa4409b0021e2800d177824f9
Submitter: Zuul
Branch: stable/queens

commit cc4f0a8fddc1fd8fa4409b0021e2800d177824f9
Author: Damien Ciabrini <email address hidden>
Date: Thu Sep 13 19:06:43 2018 -0400

    mysql: fix root password update for containerized mysql

    Since the mysql service has been containerized, we lost the ability
    to update the root password during a stack update.

    When the mysql root password in hiera differs from the one currently
    set in the mysql DB, connect to the DB with password from .my.cnf and
    update credentials of the root user before the puppet mysql module
    tries to access the database. Also update other root DB users.

    Change-Id: I8fe9a640ba36288a1f9cb18563b363159d4731c0
    Depends-On: I5bdbc89897a6dcd5bd57f2132e2acf99702b28ea
    Closes-Bug: #1792416
    (cherry picked from commit 467c6879d62b27225e3cb57a237491cd3505d92b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 8.4.1 release.