/root/.my.cnf not always created during step2 mysql_init_bundle

[James Slagle]

My initial deployment first failed during step2 which appeared to be due to pacemaker timeout.

Now when I rerun the deployment, I'm getting access denied errors from mysql during the db creates.

Seems to be due to /root/.my.cnf no longer existing under /var/lib/config-data/puppet-generated/mysql

Appears to be related to this patch:
https://review.openstack.org/#/c/602499

If I revert that from my environment and rerun mysql_init_bundle during step2, then /root/.my.cnf gets created.

[James Slagle (james-slagle)]

Just removing this line that was added by the patch seems work as well:

File['/root/.my.cnf'] -> Mysql_user<|title!='root@localhost'|>

Not sure why it doesn't get created with that line.

[James Slagle (james-slagle)]

comment 2 may not be accurate, I actually had to revert the whole patch in my environment to get it to eventually finish.

[Damien Ciabrini (dciabrin)]

James, I'm not sure what's going on yet :(

Originally the patch relied on the fact that /root/.my.cnf gets created by docker-puppet-mysql during the config generation step, so before we attempt to start pacemaker (before even step 1 in fact). Then, mysql_init_bundle bind-mounts this file from the host, so the theory was that this file would always exist when mysql_init_bundle runs puppet code.

Did you try with a non-HA deployment or with a HA deployment?

[Damien Ciabrini (dciabrin)]

Answering to myself, I wanted to asked whether you did see that behaviour only in HA deployment, or also in non-HA deployment?

I'd like to figure out how I could reproduce it locally

[James Slagle (james-slagle)]

It's an HA deployment with the standard roles using deployed server (pre-provisioned nodes).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/633768

[Damien Ciabrini (dciabrin)]

So I think a likely cause of the failure reported in the bug is that when the first stack deploy failed, another stack deploy was ran, and Heat considered it a stack UPDATE. The current logics that handles passwords in docker-puppet is broken if a stack UPDATE runs while no passwords are present on the host.

This also impact other scenario like controller replacement or major upgrades with node reprovisionning, where a new controller node is added in the stack and thus has no password yet on the filesystem.

I've just proposed a fix and I'm still testing it for both HA and non-HA deployments.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633768
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9
Submitter: Zuul
Branch: master

commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/635975

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635975
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c8b4fd25faa4241aaca658f5243509795e8a8353
Submitter: Zuul
Branch: stable/rocky

commit c8b4fd25faa4241aaca658f5243509795e8a8353
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145
    (cherry picked from commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637559

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637559
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4c51665b36240ec4a5fcfd7b275e6a37bab6673a
Submitter: Zuul
Branch: stable/queens

commit 4c51665b36240ec4a5fcfd7b275e6a37bab6673a
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145
    (cherry picked from commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.