keepalived not upgraded when going from train to ussuri

[Will Szumski]

I did an upgrade from train to ussuri via kayobe with:

kayobe overcloud service upgrade

I saw the old train container hanging around:

[centos@kayobe-aio ~]$ sudo docker ps | grep 9.3.1.6-1
53219d352c1f 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-libvirt-exporter:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_libvirt_exporter
d78ee6384ee3 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-jiralert:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_jiralert
f3dba9981adb 10.60.150.1:4000/rse-cambridge/centos-binary-prometheus-mtail:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days prometheus_mtail
36d1804256c0 10.60.150.1:4000/rse-cambridge/centos-binary-keepalived:9.3.1.6-1-centos8 "dumb-init --single-…" 2 days ago Up 2 days

Running `kayobe overcloud service deploy --skip-prechecks -t none -kt haproxy` upgraded the container. I thought this looked more like a kolla-ansible bug than a kayobe bug, so reporting here.

I am running a few patches on top of current stable/ussuri. So it is entirely possible that I broke something too.

[Will Szumski (willjs)]

I just checked an I don't have any notable changes to the haproxy rule, so assume this bug is upstream.

[Will Szumski (willjs)]

Noticed that we now call check-containers as part of upgrade, so this is probably no longer an issue on master. In ussuri it seems like to rely on a change in config.yml to restart the container. When there are no changes to the config files it doesn't get triggered. I'm running on an environment with a single host, but it seems like backup keepalived instances would be restarted due to:

https://github.com/openstack/kolla-ansible/blob/stable/ussuri/ansible/roles/haproxy/tasks/upgrade.yml#L15

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/791694

[Pierre Riteau (priteau)]

I confirm that I am seeing this on another deployment. Backup keepalived containers got updated, but not the primary one.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/791694
Committed: https://opendev.org/openstack/kolla-ansible/commit/2c728619301240c6f52894a4d0b48d0d888eaa81
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 2c728619301240c6f52894a4d0b48d0d888eaa81
Author: Mark Goddard <email address hidden>
Date: Mon May 17 09:32:40 2021 +0100

    Ensure keepalived is upgraded

    Ussuri & earlier only

    After upgrading from Train to Ussuri, if the keepalived configuration is
    unchanged, it is possible that the primary keepalived container will not
    be upgraded. This happens because we do not import check-containers.yml
    in upgrade.yml, meaning that the 'Restart keepalived container' handler
    does not fire. This change fixes the issue.

    Closes-Bug: #1928362

    Change-Id: I56775f1c0a8849c10ad5181cde6b50b2694a0512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/792517

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/792517
Committed: https://opendev.org/openstack/kolla-ansible/commit/26d310eea3f1c83dc4eb8f5806f9aad55d4a123e
Submitter: "Zuul (22348)"
Branch: stable/train

commit 26d310eea3f1c83dc4eb8f5806f9aad55d4a123e
Author: Mark Goddard <email address hidden>
Date: Mon May 17 09:32:40 2021 +0100

    Ensure keepalived is upgraded

    Ussuri & earlier only

    After upgrading from Train to Ussuri, if the keepalived configuration is
    unchanged, it is possible that the primary keepalived container will not
    be upgraded. This happens because we do not import check-containers.yml
    in upgrade.yml, meaning that the 'Restart keepalived container' handler
    does not fire. This change fixes the issue.

    Closes-Bug: #1928362

    Change-Id: I56775f1c0a8849c10ad5181cde6b50b2694a0512
    (cherry picked from commit 2c728619301240c6f52894a4d0b48d0d888eaa81)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.3.0 release.