Major upgrade of HA services doesn't update container image

[Damien Ciabrini]

Since Queens, major upgrade of containerized HA services is implemented in upgrade tasks.

While it is possible to change the image tag during a major upgrade [1], changing the entire image name is currently not working. This is because the pacemaker bundle themselves are never updated to use the new images.

[1] I85e11dd93c7fd2c42e71b467f46b0044d4516524

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/560408

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/560426

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/560408
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e8a1fc25d77b89ad82e854e1f19a247d3d990f2b
Submitter: Zuul
Branch: master

commit e8a1fc25d77b89ad82e854e1f19a247d3d990f2b
Author: Damien Ciabrini <email address hidden>
Date: Wed Apr 11 11:57:59 2018 +0000

    Upgrade: make bundles use new container image name after upgrade

    The major_upgrade tasks for HA services only allows to change the container
    image tag used by bundles. It doesn't work when the image name changes.

    Fix this unwanted behaviour by updating the bundle's attribute in pacemaker
    to use container image <NEW>:pcmklatest instead of <CURRENT>:pcmklatest
    We are constrained by the steps at when we can modify the bundle:
      . Image update must stay at step 3 when pacemaker is stopped.
      . image name used by the bundle must be available in docker when the
        bundle is updated

    So we re-use the double tagging idiom to perform the image update:
      . At step 0, we tag the image pointed to by <CURRENT>:pcmklatest
        with an additional temporary tag <NEW>:pcmklatest.
        => this ensures that at step1, the new tag is available on all
           controller nodes.
      . At step 1, we update the resource bundle to use the new image
        name <NEW>:pcmklatest.
        => at the end of step1, the bundle will be configured with the
           new name, and be able to start even if the real container
           image hasn't be pulled yet.
      . At step 3, the existing code will download the real image
        <NEW>:<NEWTAG> and make tag <NEW>:pcmklatest point to it.

    Since the bundle is always modified, we now stop and restart the
    bundle resources unconditionally.

    Also, move the mariadb upgrade task to step 3, when pacemaker is
    guaranteed to be stopped, because the task assumes that no mysql
    is running while it runs. Fix the mysql permission after rpm
    upgrade on the host.

    Change-Id: Ic87a66753b104b9f15db70fdccbd66d88cef94df
    Closes-Bug: #1763001

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/563588

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563073
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8530dd9ddca1b15de6be9343e63fd1e96493cd44
Submitter: Zuul
Branch: master

commit 8530dd9ddca1b15de6be9343e63fd1e96493cd44
Author: mandreou <email address hidden>
Date: Fri Apr 20 13:39:56 2018 +0300

    Make pcs resource bundle image name update tolerant of rerun

    The parent review at Ic87a66753b104b9f15db70fdccbd66d88cef94df
    allows us to update the name for pcs resource bundle resources
    if this is changed as part of the upgrade configuration.

    If the upgrade is interrupted the target pacemaker resource bundle
    may not even have been created yet. This groups stop/update/start
    the bundle resource and adds a new conditional to check if the
    cluster resource exists before trying to update the container
    image being used. Otherwise a re-run of the upgrade tasks may
    fail if the cluster resource doesn't exist.

    Related-Bug: 1763001
    Change-Id: Ifc6f78d73bc71a5b5edfadfbfacaa3560fe7c2df

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563588
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=12a10617ad53830ea5e390fbbcbf96e0bd1e6a9d
Submitter: Zuul
Branch: stable/queens

commit 12a10617ad53830ea5e390fbbcbf96e0bd1e6a9d
Author: mandreou <email address hidden>
Date: Fri Apr 20 13:39:56 2018 +0300

    Make pcs resource bundle image name update tolerant of rerun

    The parent review at Ic87a66753b104b9f15db70fdccbd66d88cef94df
    allows us to update the name for pcs resource bundle resources
    if this is changed as part of the upgrade configuration.

    If the upgrade is interrupted the target pacemaker resource bundle
    may not even have been created yet. This groups stop/update/start
    the bundle resource and adds a new conditional to check if the
    cluster resource exists before trying to update the container
    image being used. Otherwise a re-run of the upgrade tasks may
    fail if the cluster resource doesn't exist.

    Related-Bug: 1763001
    (cherry picked from commit 8530dd9ddca1b15de6be9343e63fd1e96493cd44)
    Change-Id: Ifc6f78d73bc71a5b5edfadfbfacaa3560fe7c2df

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b3 development milestone.