Injecting certificate with "podman cp" can break cluster monitoring and operation

[Damien Ciabrini]

When internal or public certificates are re-generated, we notify the
impacted services by triggering a "config reload" when possible, to
avoid restarting them entirely and incurring a temporary service
disruption.

Since the certificates are re-generated on the host, they must be
injected in running containers before the config reload takes place.
We inject the file into the running container with "podman cp".

"podman cp" internally uses "podman pause", which freezes the
container execution. While the container is frozen, any attempt at
running "podman exec" or "podman stop" concurrently will fail with
a generic error:

Feb 24 02:05:58 controller-0 podman(haproxy-bundle-podman-0)[352861]: ERROR: Error: can only stop created or running containers. 75ee35ce1b0912645d42d46967a372cf0a3edf49a5d9d5d121ce60fe3acc6144 is in state paused: container state improper
Feb 24 02:05:58 controller-0 podman(haproxy-bundle-podman-0)[352861]: ERROR: Failed to stop container, haproxy-bundle-podman-0, based on image, cluster.common.tag/rhosp16-openstack-haproxy:pcmklatest.

During this time window, it may happen that pacemaker cannot monitor the
state of a container and decide to stop it. Worse, it the "podman
stop" cannot be run, pacemaker will consider it a stop failure and
will end up fencing the node hosting the running container.

[Luigi Toscano (ltoscano)]

Fix on master: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/782539

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/783942
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/63001263ad011d5a1dcca42fc7c79599fe6c78c8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 63001263ad011d5a1dcca42fc7c79599fe6c78c8
Author: Damien Ciabrini <email address hidden>
Date: Mon Mar 22 18:04:08 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    (cherry-picked from 93e53b74293cb4478ea415255fee96e7fddda004)
    (squashed with Ic6e4264c5ad46bd2589cc907c365af2d42fde63d)
    (removed a part that should stay in puppet-tripleo before wallaby)

    Closes-Bug: #1917868

    Change-Id: Id7308f028f33716be5e3df6699c3f2c12e33e344

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/783901
Committed: https://opendev.org/openstack/puppet-tripleo/commit/f6c88d0146a0ac35944433b11673f0bc036051ff
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f6c88d0146a0ac35944433b11673f0bc036051ff
Author: Damien Ciabrini <email address hidden>
Date: Fri Mar 5 13:26:45 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    Adapted from Id7308f028f33716be5e3df6699c3f2c12e33e344, as the
    same behaviour is implemented in puppet-tripleo before wallaby.

    Change-Id: I14be16052677bf3426a88ec4b5299f9502007472
    Related-Bug: #1917868

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.1.0 release.

[Luigi Toscano (ltoscano)]

See the previous comment: this is released as part of openstack/tripleo-heat-templates 14.1.0. Retargeting and fixing the status.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/789875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790174

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/783913
Committed: https://opendev.org/openstack/puppet-tripleo/commit/49728d24087ef5b3a07eba288bccbbc16f6717af
Submitter: "Zuul (22348)"
Branch: stable/train

commit 49728d24087ef5b3a07eba288bccbbc16f6717af
Author: Damien Ciabrini <email address hidden>
Date: Fri Mar 5 13:26:45 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    Adapted from Id7308f028f33716be5e3df6699c3f2c12e33e344, as the
    same behaviour is implemented in puppet-tripleo before wallaby.

    Change-Id: I14be16052677bf3426a88ec4b5299f9502007472
    Related-Bug: #1917868

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/783949
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ab5d866cbc8bd61e04010611b028f9d20292bbe5
Submitter: "Zuul (22348)"
Branch: stable/train

commit ab5d866cbc8bd61e04010611b028f9d20292bbe5
Author: Damien Ciabrini <email address hidden>
Date: Mon Mar 22 18:04:08 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    (cherry-picked from 93e53b74293cb4478ea415255fee96e7fddda004)
    (squashed with Ic6e4264c5ad46bd2589cc907c365af2d42fde63d)
    (removed a part that should stay in puppet-tripleo before wallaby)

    Closes-Bug: #1917868

    Change-Id: Id7308f028f33716be5e3df6699c3f2c12e33e344

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/789875
Committed: https://opendev.org/openstack/puppet-tripleo/commit/e09d2a192c888aa84f288a4742e81bab74067dff
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit e09d2a192c888aa84f288a4742e81bab74067dff
Author: Damien Ciabrini <email address hidden>
Date: Fri Mar 5 13:26:45 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    Adapted from Id7308f028f33716be5e3df6699c3f2c12e33e344, as the
    same behaviour is implemented in puppet-tripleo before wallaby.

    Change-Id: I14be16052677bf3426a88ec4b5299f9502007472
    Related-Bug: #1917868
    (cherry picked from commit f6c88d0146a0ac35944433b11673f0bc036051ff)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.6.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790174
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f43f11f7dcbaa077b1b1934473c9143f0e89f90d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f43f11f7dcbaa077b1b1934473c9143f0e89f90d
Author: Damien Ciabrini <email address hidden>
Date: Mon Mar 22 18:04:08 2021 +0100

    HA: inject public certificates without blocking container

    Do not inject public certificates in pacemaker bundles by means
    of "podman cp", as this pauses the container for a short amount
    of time and can make pacemaker operation fail during that time
    window and impact cluster for no reason.

    Keep "podman cp" for non-HA containers, as the freeze is short
    and doesn't seem to impact podman monitoring anyway.

    The new certificate injection only works for podman 1.9+, lower
    version won't overwrite the existing certificate.

    (cherry-picked from 93e53b74293cb4478ea415255fee96e7fddda004)
    (squashed with Ic6e4264c5ad46bd2589cc907c365af2d42fde63d)
    (removed a part that should stay in puppet-tripleo before wallaby)

    Closes-Bug: #1917868

    Change-Id: Id7308f028f33716be5e3df6699c3f2c12e33e344
    (cherry picked from commit 63001263ad011d5a1dcca42fc7c79599fe6c78c8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.5 release.

[Bogdan Dobrelya (bogdando)]

Stdin piping is broken [0] for podman 1.6.4, we should revert https://review.opendev.org/c/openstack/tripleo-heat-templates/+/783949/

[0] https://github.com/containers/podman/issues/5046

[Bogdan Dobrelya (bogdando)]

please disregard, upstream revert would flow to 16.2, not to 16.1. So we cannot revert at this time and place.

[Luigi Toscano (ltoscano)]

As far as RDO/train/centos8* is concerned (where IIRC podman 3.x is available) everything should be fine and awesome as it is right now.