Queens: Updating mariadb-server package on the host disrupts containerized mysql service

[Damien Ciabrini]

Spotted during a minor update of an HA overcloud.

If mariadb-server rpms are installed on the host, and packages mariadb-server* can be upgraded during the update_tasks, the rpm scriptlets will change the ownership of /var/lib/mysql to mysql user from the host (uid=27) in place of kolla's mysql user (uid=42434).

This confuses the containerized server, pacemaker detects it and tries to restart it, without being able to do so, because ownership is not changed unless the entire container is restarted.

Failed Resource Actions:
* rabbitmq_monitor_10000 on rabbitmq-bundle-2 'not running' (7): call=32, status=complete, exitreason='',
    last-rc-change='Thu Oct 7 15:30:02 2021', queued=0ms, exec=0ms
* galera_start_0 on galera-bundle-0 'insufficient privileges' (4): call=12, status=complete, exitreason='Directory /var/lib/mysql is not writable by mysql',
    last-rc-change='Thu Oct 7 10:58:03 2021', queued=0ms, exec=604ms

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814532

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815775

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815792

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815793

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815794

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815775
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d33865cded7766eb8d35ce56d5eb206c2dda1ad9
Submitter: "Zuul (22348)"
Branch: master

commit d33865cded7766eb8d35ce56d5eb206c2dda1ad9
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 20 15:49:52 2021 +0200

    Remove mariadb-server packages from the host

    During updates/upgrades, installing mariadb-server on the
    host may impact the behaviour of containerized mysql.

    During a FFU if mariadb-server is upgraded, it may happen
    that the rpm scriptlets fail to start mariadb server and
    leave a crash log behind (tc.log). This prevents the
    regular online upgrade from happening.

    During an upgrade, mariadb-server used to be force-upgraded
    in the mysql service for historical reasons. It's not
    necessary since mysql is containerized and can trigger
    the same crash as explained above.

    During a minor update, the same reasoning can apply if
    RHEL channel ships a new mariadb-server rpm as scriptlets
    will probably leave a crash behind as well.

    Make sure mariadb-server is never installed, while
    keeping mariadb CLI if already present on the host, to
    avoid operational impacts.

    Change-Id: Ib669bb4a5fcbb493d6d5edb5999bd1d87418558b
    Closes-Bug: #1946742

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815792
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/4bf7f940b13f95ee331b638b8070a03d4d9c014d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4bf7f940b13f95ee331b638b8070a03d4d9c014d
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 20 15:49:52 2021 +0200

    Remove mariadb-server packages from the host

    During updates/upgrades, installing mariadb-server on the
    host may impact the behaviour of containerized mysql.

    During a FFU if mariadb-server is upgraded, it may happen
    that the rpm scriptlets fail to start mariadb server and
    leave a crash log behind (tc.log). This prevents the
    regular online upgrade from happening.

    During an upgrade, mariadb-server used to be force-upgraded
    in the mysql service for historical reasons. It's not
    necessary since mysql is containerized and can trigger
    the same crash as explained above.

    During a minor update, the same reasoning can apply if
    RHEL channel ships a new mariadb-server rpm as scriptlets
    will probably leave a crash behind as well.

    Make sure mariadb-server is never installed, while
    keeping mariadb CLI if already present on the host, to
    avoid operational impacts.

    Change-Id: Ib669bb4a5fcbb493d6d5edb5999bd1d87418558b
    Closes-Bug: #1946742
    (cherry picked from commit d33865cded7766eb8d35ce56d5eb206c2dda1ad9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815793
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/1a20780ae062d1ac6e992623917dc5eac8cf65c7
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 1a20780ae062d1ac6e992623917dc5eac8cf65c7
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 20 15:49:52 2021 +0200

    Remove mariadb-server packages from the host

    During updates/upgrades, installing mariadb-server on the
    host may impact the behaviour of containerized mysql.

    During a FFU if mariadb-server is upgraded, it may happen
    that the rpm scriptlets fail to start mariadb server and
    leave a crash log behind (tc.log). This prevents the
    regular online upgrade from happening.

    During an upgrade, mariadb-server used to be force-upgraded
    in the mysql service for historical reasons. It's not
    necessary since mysql is containerized and can trigger
    the same crash as explained above.

    During a minor update, the same reasoning can apply if
    RHEL channel ships a new mariadb-server rpm as scriptlets
    will probably leave a crash behind as well.

    Make sure mariadb-server is never installed, while
    keeping mariadb CLI if already present on the host, to
    avoid operational impacts.

    Change-Id: Ib669bb4a5fcbb493d6d5edb5999bd1d87418558b
    Closes-Bug: #1946742
    (cherry picked from commit d33865cded7766eb8d35ce56d5eb206c2dda1ad9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/815794
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ae93eba463f77ee50d6530658beee9fa09d4093b
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ae93eba463f77ee50d6530658beee9fa09d4093b
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 20 15:49:52 2021 +0200

    Remove mariadb-server packages from the host

    During updates/upgrades, installing mariadb-server on the
    host may impact the behaviour of containerized mysql.

    During a FFU if mariadb-server is upgraded, it may happen
    that the rpm scriptlets fail to start mariadb server and
    leave a crash log behind (tc.log). This prevents the
    regular online upgrade from happening.

    During an upgrade, mariadb-server used to be force-upgraded
    in the mysql service for historical reasons. It's not
    necessary since mysql is containerized and can trigger
    the same crash as explained above.

    During a minor update, the same reasoning can apply if
    RHEL channel ships a new mariadb-server rpm as scriptlets
    will probably leave a crash behind as well.

    Make sure mariadb-server is never installed, while
    keeping mariadb CLI if already present on the host, to
    avoid operational impacts.

    Change-Id: Ib669bb4a5fcbb493d6d5edb5999bd1d87418558b
    Closes-Bug: #1946742
    (cherry picked from commit d33865cded7766eb8d35ce56d5eb206c2dda1ad9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814786
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/8c91e229b1a846b70029ed746e362482026bfc23
Submitter: "Zuul (22348)"
Branch: stable/train

commit 8c91e229b1a846b70029ed746e362482026bfc23
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 20 15:49:52 2021 +0200

    Remove mariadb-server packages from the host

    During updates/upgrades, installing mariadb-server on the
    host may impact the behaviour of containerized mysql.

    During a FFU if mariadb-server is upgraded, it may happen
    that the rpm scriptlets fail to start mariadb server and
    leave a crash log behind (tc.log). This prevents the
    regular online upgrade from happening.

    During an upgrade, mariadb-server used to be force-upgraded
    in the mysql service for historical reasons. It's not
    necessary since mysql is containerized and can trigger
    the same crash as explained above.

    During a minor update, the same reasoning can apply if
    RHEL channel ships a new mariadb-server rpm as scriptlets
    will probably leave a crash behind as well.

    Make sure mariadb-server is never installed, while
    keeping mariadb CLI if already present on the host, to
    avoid operational impacts.

    Change-Id: Ib669bb4a5fcbb493d6d5edb5999bd1d87418558b
    Closes-Bug: #1946742
    (cherry picked from commit d33865cded7766eb8d35ce56d5eb206c2dda1ad9)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Damien Ciabrini <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814532

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.