ephemeral heat renders minor update code paths non functional

[Michele Baldessari]

With ephemeral heat the stack status is always CREATE. Since we leverage that information in the pacemaker_restart_bundle script, to determine if this is a fresh deployment vs a redeploy (aka stack UPDATE) we need to come up with a different mechanism.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/806610
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b8d9370997ee6f910d401399f72f97272d639c73
Submitter: "Zuul (22348)"
Branch: master

commit b8d9370997ee6f910d401399f72f97272d639c73
Author: Damien Ciabrini <email address hidden>
Date: Mon Aug 30 17:17:05 2021 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, we simply check if the
    resource existed before running puppet. If it did not, we're part of
    an initial deployment, if it did exist then this is a redeploy and
    in such case, if the config changed we will restart the service on
    the bootstrap node.

    We also cleanup the molecule steps as there are some redundancies there.

    Testing protocol is in the parent change.

    Related-Bug: #1942309

    Change-Id: I08cff959d7d104ad0e8d833ed6ce06df8ff38b16

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/807761

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/806611
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ad2a13ab47b6ad6c948d8c229dde8b9f1386ea29
Submitter: "Zuul (22348)"
Branch: master

commit ad2a13ab47b6ad6c948d8c229dde8b9f1386ea29
Author: Damien Ciabrini <email address hidden>
Date: Mon Aug 30 17:17:46 2021 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, in ansible we added a
    TRIPLEO_HA_WRAPPER_RESOURCE_EXISTS env variable which will be true
    when the resource existed even before calling puppet.

    This way we can restore the previous behaviour (which was relying
    on the stack_update hiera key) of restarting an HA
    bundle on the bootstrap node in case of a configuration change.

    While we're at it we make sure that the logging takes place via logger
    so we're sure to capture these events in the journal.

    Tested as follows:
    1) Initial deploy:
    [root@controller-0 ~]# journalctl |grep pcmkres
    Sep 01 10:23:35 controller-0.alejandro.ftw pcmkrestart[47636]: Initial deployment, skipping the restart of haproxy-bundle
    Sep 01 10:24:25 controller-0.alejandro.ftw pcmkrestart[49735]: Initial deployment, skipping the restart of galera-bundle
    Sep 01 10:25:15 controller-0.alejandro.ftw pcmkrestart[53052]: Initial deployment, skipping the restart of rabbitmq-bundle
    Sep 01 10:37:35 controller-0.alejandro.ftw pcmkrestart[148651]: Initial deployment, skipping the restart of openstack-cinder-volume

    Redeploy changing only the haproxy config via a hiera key change:
    Sep 01 11:12:29 controller-0.alejandro.ftw pcmkrestart[438507]: Wed Sep Restarting haproxy-bundle globally. Stopping:
    Sep 01 11:12:37 controller-0.alejandro.ftw pcmkrestart[439271]: Wed Sep Restarting haproxy-bundle globally. Starting:

    Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/806610/

    Closes-Bug: #1942309

    Change-Id: I90ea2287b5ab32c8dc6bbf5f91927e7488326dcd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/807861

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/807761
Committed: https://opendev.org/openstack/tripleo-ansible/commit/39429a65166d5e8f7ca26102a90cef7236fe031c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 39429a65166d5e8f7ca26102a90cef7236fe031c
Author: Damien Ciabrini <email address hidden>
Date: Mon Aug 30 17:17:05 2021 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, we simply check if the
    resource existed before running puppet. If it did not, we're part of
    an initial deployment, if it did exist then this is a redeploy and
    in such case, if the config changed we will restart the service on
    the bootstrap node.

    We also cleanup the molecule steps as there are some redundancies there.

    Testing protocol is in the parent change.

    Related-Bug: #1942309

    Change-Id: I08cff959d7d104ad0e8d833ed6ce06df8ff38b16
    (cherry picked from commit b8d9370997ee6f910d401399f72f97272d639c73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/807861
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/79791ce30d3203d03e9a4db4b3c18991d5102ff2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 79791ce30d3203d03e9a4db4b3c18991d5102ff2
Author: Damien Ciabrini <email address hidden>
Date: Mon Aug 30 17:17:46 2021 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, in ansible we added a
    TRIPLEO_HA_WRAPPER_RESOURCE_EXISTS env variable which will be true
    when the resource existed even before calling puppet.

    This way we can restore the previous behaviour (which was relying
    on the stack_update hiera key) of restarting an HA
    bundle on the bootstrap node in case of a configuration change.

    While we're at it we make sure that the logging takes place via logger
    so we're sure to capture these events in the journal.

    Tested as follows:
    1) Initial deploy:
    [root@controller-0 ~]# journalctl |grep pcmkres
    Sep 01 10:23:35 controller-0.alejandro.ftw pcmkrestart[47636]: Initial deployment, skipping the restart of haproxy-bundle
    Sep 01 10:24:25 controller-0.alejandro.ftw pcmkrestart[49735]: Initial deployment, skipping the restart of galera-bundle
    Sep 01 10:25:15 controller-0.alejandro.ftw pcmkrestart[53052]: Initial deployment, skipping the restart of rabbitmq-bundle
    Sep 01 10:37:35 controller-0.alejandro.ftw pcmkrestart[148651]: Initial deployment, skipping the restart of openstack-cinder-volume

    Redeploy changing only the haproxy config via a hiera key change:
    Sep 01 11:12:29 controller-0.alejandro.ftw pcmkrestart[438507]: Wed Sep Restarting haproxy-bundle globally. Stopping:
    Sep 01 11:12:37 controller-0.alejandro.ftw pcmkrestart[439271]: Wed Sep Restarting haproxy-bundle globally. Starting:

    Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/807761

    Closes-Bug: #1942309

    Change-Id: I90ea2287b5ab32c8dc6bbf5f91927e7488326dcd
    (cherry picked from commit ad2a13ab47b6ad6c948d8c229dde8b9f1386ea29)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/836134

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836171

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/836134
Committed: https://opendev.org/openstack/tripleo-ansible/commit/be5663532e105ab72279db4351efd45d1406ef16
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit be5663532e105ab72279db4351efd45d1406ef16
Author: Damien Ciabrini <email address hidden>
Date: Fri Apr 1 12:47:28 2022 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, we simply check if the
    resource existed before running puppet. If it did not, we're part of
    an initial deployment, if it did exist then this is a redeploy and
    in such case, if the config changed we will restart the service on
    the bootstrap node.

    We also cleanup the molecule steps as there are some redundancies there.

    Testing protocol is in the parent change.

    Note: manually squashed I5199cf5fd20590a2aa1af5c52ef805067f995366
    to fix restart of all HA resources without stack_action.

    Related-Bug: #1942309
    Change-Id: I08cff959d7d104ad0e8d833ed6ce06df8ff38b16
    (cherry picked from commit b8d9370997ee6f910d401399f72f97272d639c73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836171
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fb88ed98a2e4b46161ab5193d03338c3eb6b306a
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit fb88ed98a2e4b46161ab5193d03338c3eb6b306a
Author: Damien Ciabrini <email address hidden>
Date: Fri Apr 1 13:37:20 2022 +0200

    Check whether an HA resource already exists explicitly

    With ephemeral heat we lost the meaning of the 'stack_action' hiera key
    which we previously used to distinguish between fresh deployment and
    pre-existing deployment (aka redeploy).
    Since this hiera key is not available anymore, in ansible we added a
    TRIPLEO_HA_WRAPPER_RESOURCE_EXISTS env variable which will be true
    when the resource existed even before calling puppet.

    This way we can restore the previous behaviour (which was relying
    on the stack_update hiera key) of restarting an HA
    bundle on the bootstrap node in case of a configuration change.

    While we're at it we make sure that the logging takes place via logger
    so we're sure to capture these events in the journal.

    Tested as follows:
    1) Initial deploy:
    [root@controller-0 ~]# journalctl |grep pcmkres
    Sep 01 10:23:35 controller-0.alejandro.ftw pcmkrestart[47636]: Initial deployment, skipping the restart of haproxy-bundle
    Sep 01 10:24:25 controller-0.alejandro.ftw pcmkrestart[49735]: Initial deployment, skipping the restart of galera-bundle
    Sep 01 10:25:15 controller-0.alejandro.ftw pcmkrestart[53052]: Initial deployment, skipping the restart of rabbitmq-bundle
    Sep 01 10:37:35 controller-0.alejandro.ftw pcmkrestart[148651]: Initial deployment, skipping the restart of openstack-cinder-volume

    Redeploy changing only the haproxy config via a hiera key change:
    Sep 01 11:12:29 controller-0.alejandro.ftw pcmkrestart[438507]: Wed Sep Restarting haproxy-bundle globally. Stopping:
    Sep 01 11:12:37 controller-0.alejandro.ftw pcmkrestart[439271]: Wed Sep Restarting haproxy-bundle globally. Starting:

    Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/836134

    Closes-Bug: #1942309

    Change-Id: I90ea2287b5ab32c8dc6bbf5f91927e7488326dcd
    (cherry picked from commit ad2a13ab47b6ad6c948d8c229dde8b9f1386ea29)
    (cherry picked from commit 79791ce30d3203d03e9a4db4b3c18991d5102ff2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.6.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/876927

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/876929

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/876929
Reason: Train is about to transition to End of Life. Open patches needs to be abandoned before branch deletion.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/876927
Reason: Train is about to transition to End of Life. Open patches needs to be abandoned before branch deletion.