/root/.my.cnf not always created during step2 mysql_init_bundle

Bug #1809145 reported by James Slagle on 2018-12-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Damien Ciabrini

Bug Description

My initial deployment first failed during step2 which appeared to be due to pacemaker timeout.

Now when I rerun the deployment, I'm getting access denied errors from mysql during the db creates.

Seems to be due to /root/.my.cnf no longer existing under /var/lib/config-data/puppet-generated/mysql

Appears to be related to this patch:
https://review.openstack.org/#/c/602499

If I revert that from my environment and rerun mysql_init_bundle during step2, then /root/.my.cnf gets created.

James Slagle (james-slagle) wrote :
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → stein-3
James Slagle (james-slagle) wrote :

Just removing this line that was added by the patch seems work as well:

File['/root/.my.cnf'] -> Mysql_user<|title!='root@localhost'|>

Not sure why it doesn't get created with that line.

James Slagle (james-slagle) wrote :

comment 2 may not be accurate, I actually had to revert the whole patch in my environment to get it to eventually finish.

Damien Ciabrini (dciabrin) wrote :

James, I'm not sure what's going on yet :(

Originally the patch relied on the fact that /root/.my.cnf gets created by docker-puppet-mysql during the config generation step, so before we attempt to start pacemaker (before even step 1 in fact). Then, mysql_init_bundle bind-mounts this file from the host, so the theory was that this file would always exist when mysql_init_bundle runs puppet code.

Did you try with a non-HA deployment or with a HA deployment?

Changed in tripleo:
assignee: nobody → Damien Ciabrini (dciabrin)
Damien Ciabrini (dciabrin) wrote :

Answering to myself, I wanted to asked whether you did see that behaviour only in HA deployment, or also in non-HA deployment?

I'd like to figure out how I could reproduce it locally

James Slagle (james-slagle) wrote :

It's an HA deployment with the standard roles using deployed server (pre-provisioned nodes).

Fix proposed to branch: master
Review: https://review.openstack.org/633768

Changed in tripleo:
status: Triaged → In Progress
tags: added: pike-backport-potential queens-backport-potential rocky-backport-potential
tags: added: idempotency
Damien Ciabrini (dciabrin) wrote :

So I think a likely cause of the failure reported in the bug is that when the first stack deploy failed, another stack deploy was ran, and Heat considered it a stack UPDATE. The current logics that handles passwords in docker-puppet is broken if a stack UPDATE runs while no passwords are present on the host.

This also impact other scenario like controller replacement or major upgrades with node reprovisionning, where a new controller node is added in the stack and thus has no password yet on the filesystem.

I've just proposed a fix and I'm still testing it for both HA and non-HA deployments.

Reviewed: https://review.openstack.org/633768
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9
Submitter: Zuul
Branch: master

commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/635975
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c8b4fd25faa4241aaca658f5243509795e8a8353
Submitter: Zuul
Branch: stable/rocky

commit c8b4fd25faa4241aaca658f5243509795e8a8353
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145
    (cherry picked from commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9)

tags: added: in-stable-rocky

Reviewed: https://review.openstack.org/637559
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4c51665b36240ec4a5fcfd7b275e6a37bab6673a
Submitter: Zuul
Branch: stable/queens

commit 4c51665b36240ec4a5fcfd7b275e6a37bab6673a
Author: Damien Ciabrini <email address hidden>
Date: Tue Jan 29 14:49:46 2019 +0000

    Fix generation of configs that contain password files

    In I8fe9a640ba36288a1f9cb18563b363159d4731c0 we added the ability
    to prevent overwriting password files during docker-puppet runs, to
    give the service the ability to update his own user credentials.

    This doesn't work in case a stack update is running and config files
    don't exist on the host in the first place (e.g. because of a
    previous deploy failure, or due to a controller node replacement).
    This also causes complications if a password file is already present
    during a stack creation (e.g re-creating a stack on a split-stack
    environment).

    Change the way password files are handled:

      . if a previous password file exists on the host, do not overwrite
        it with the new password. Only use the new password for
        computing the hash.

      . otherwise, always copy the newly generated password file on the
        host.

    Also, fix the config hash generation that currently considers the
    password file twice, which makes the hash vary and cause
    unexpected service restart at each stack update.

    Change-Id: Ia77f1a82c4164f53fa90a6f05ba728787622285d
    Closes-bug: #1809145
    (cherry picked from commit ec5fbe8de78ad37ec8ed4f6642be0e545da7fcf9)

tags: added: in-stable-queens

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers