Overcloud update converge fails when updating Octavia to latest Queens, Rocky, Stein

[Carlos Goncalves]

Steps to Reproduce:
1. Deploy TripleO Stein prior to https://github.com/openstack/tripleo-common/commit/1ecf1827d5dcacd561bd4dee65a4b772223ad63c
2. The update will fail while executing "openstack overcloud update converge" with patch above included.

Actual results:

undercloud:/var/lib/mistral/overcloud/octavia-ansible/octavia-ansible.log

2020-02-16 21:36:29,806 p=350480 u=root | TASK [octavia-controller-post-config : check if ip list needs updating] ********
2020-02-16 21:36:29,913 p=350480 u=root | ok: [controller-0]
2020-02-16 21:36:30,009 p=350480 u=root | ok: [controller-1]
2020-02-16 21:36:30,026 p=350480 u=root | ok: [controller-2]
2020-02-16 21:36:30,116 p=350480 u=root | TASK [octavia-controller-post-config : setting [health_manager]/controller_ip_port_list] ***
2020-02-16 21:36:30,790 p=350480 u=root | changed: [controller-0]
2020-02-16 21:36:30,912 p=350480 u=root | changed: [controller-1]
2020-02-16 21:36:30,973 p=350480 u=root | changed: [controller-2]
2020-02-16 21:36:31,065 p=350480 u=root | TASK [octavia-controller-post-config : create a symbolic link conf.d/post-deploy -> post-deploy.conf (LP#1836074)] ***
2020-02-16 21:36:31,754 p=350480 u=root | fatal: [controller-0]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0644", "msg": "refusing to convert from file to symlink for /var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "owner": "root", "path": "/var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "secontext": "system_u:object_r:container_file_t:s0", "size": 193, "state": "file", "uid": 0}
2020-02-16 21:36:31,863 p=350480 u=root | fatal: [controller-1]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0644", "msg": "refusing to convert from file to symlink for /var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "owner": "root", "path": "/var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "secontext": "system_u:object_r:container_file_t:s0", "size": 193, "state": "file", "uid": 0}
2020-02-16 21:36:31,920 p=350480 u=root | fatal: [controller-2]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0644", "msg": "refusing to convert from file to symlink for /var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "owner": "root", "path": "/var/lib/config-data/puppet-generated/octavia/etc/octavia/conf.d/common/post-deploy.conf", "secontext": "system_u:object_r:container_file_t:s0", "size": 193, "state": "file", "uid": 0}
2020-02-16 21:36:31,921 p=350480 u=root | PLAY RECAP *********************************************************************
2020-02-16 21:36:31,921 p=350480 u=root | controller-0 : ok=51 changed=13 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0
2020-02-16 21:36:31,921 p=350480 u=root | controller-1 : ok=37 changed=10 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
2020-02-16 21:36:31,922 p=350480 u=root | controller-2 : ok=37 changed=10 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
2020-02-16 21:36:31,922 p=350480 u=root | undercloud-0 : ok=16 changed=7 unreachable=0 failed=0 skipped=15 rescued=0 ignored=1

[Carlos Goncalves (cgoncalves)]

Reported originally at https://bugzilla.redhat.com/show_bug.cgi?id=1803708

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/708090

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/708091

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/708093

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708093
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=e93f299b0f68e488f53ddd98d47df1d61f250e05
Submitter: Zuul
Branch: stable/queens

commit e93f299b0f68e488f53ddd98d47df1d61f250e05
Author: Carlos Goncalves <email address hidden>
Date: Mon Feb 17 11:45:11 2020 +0100

    Do not create symlink conf.d/post-deploy

    This task was added to work-around a circular dependency in CI. It can
    and should now be removed as it causes overcloud update converge to
    fail.

    Closes-Bug: #1863595
    Change-Id: I66266b4a0951add722e5700a500ad10092b34ee1
    (cherry picked from commit 1fbf82b7a644e33e92de32c369802d63e04cd425)
    (cherry picked from commit ec5a85451da061c93f65b08284ab2cf644cfaa09)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708091
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=ec5a85451da061c93f65b08284ab2cf644cfaa09
Submitter: Zuul
Branch: stable/rocky

commit ec5a85451da061c93f65b08284ab2cf644cfaa09
Author: Carlos Goncalves <email address hidden>
Date: Mon Feb 17 11:45:11 2020 +0100

    Do not create symlink conf.d/post-deploy

    This task was added to work-around a circular dependency in CI. It can
    and should now be removed as it causes overcloud update converge to
    fail.

    Closes-Bug: #1863595
    Change-Id: I66266b4a0951add722e5700a500ad10092b34ee1
    (cherry picked from commit 1fbf82b7a644e33e92de32c369802d63e04cd425)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708090
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=1fbf82b7a644e33e92de32c369802d63e04cd425
Submitter: Zuul
Branch: stable/stein

commit 1fbf82b7a644e33e92de32c369802d63e04cd425
Author: Carlos Goncalves <email address hidden>
Date: Mon Feb 17 11:45:11 2020 +0100

    Do not create symlink conf.d/post-deploy

    This task was added to work-around a circular dependency in CI. It can
    and should now be removed as it causes overcloud update converge to
    fail.

    Closes-Bug: #1863595
    Change-Id: I66266b4a0951add722e5700a500ad10092b34ee1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/750986

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750986
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d3826e9ec3c0532396c5c795e5764b24e63ddfed
Submitter: Zuul
Branch: master

commit d3826e9ec3c0532396c5c795e5764b24e63ddfed
Author: Brent Eagles <email address hidden>
Date: Thu Sep 10 10:37:57 2020 -0230

    Create post-deploy.conf on update/upgrade for octavia

    Containers are restarted with new command lines during upgrade/updates
    before the external_deploy_task are run that create a configuration file
    that is used on the command line. This results in octavia services
    failing to start.

    Change-Id: I741059afad42d0aa1e17b5becd56cbbbb0003c82
    Related-Bug: #1863595

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/763426

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/763571

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common rocky-eol release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/789405

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/789406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/789406
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/2b01176450a199d37bc45c944b167c94e6beaada
Submitter: "Zuul (22348)"
Branch: stable/train

commit 2b01176450a199d37bc45c944b167c94e6beaada
Author: Brent Eagles <email address hidden>
Date: Thu Sep 10 10:37:57 2020 -0230

    Create post-deploy.conf on update/upgrade for octavia

    Containers are restarted with new command lines during upgrade/updates
    before the external_deploy_task are run that create a configuration file
    that is used on the command line. This results in octavia services
    failing to start.

    Note: this was originally merged as:
        https://review.opendev.org/#/c/750986/
    but the OctaviaBase references in templates was incorrect so the
    original patch was reverted through:
        https://review.opendev.org/#/c/763561/
    As the original patch had not been backported, it makes more sense to
    revert and get a correct fix in place and backport that instead of
    requiring backporting a series or backporting a squashed commit.

    Conflicts:
      deployment/octavia/octavia-api-container-puppet.yaml
      deployment/octavia/octavia-housekeeping-container-puppet.yaml

    Change-Id: Ib3476e53f89b50bae72b9c95a5d3dec51ed3de7e
    Related-Bug: #1863595
    (cherry picked from commit 6c11a949fc0b940ebc3815aeb86156bd60df5851)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common queens-eol release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/772390
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/a68d3f96b8351ad6b5db6e5bd8f0efcb85ffd8a9
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a68d3f96b8351ad6b5db6e5bd8f0efcb85ffd8a9
Author: Brent Eagles <email address hidden>
Date: Thu Sep 10 10:37:57 2020 -0230

    Create post-deploy.conf on update/upgrade for octavia

    Containers are restarted with new command lines during upgrade/updates
    before the external_deploy_task are run that create a configuration file
    that is used on the command line. This results in octavia services
    failing to start.

    Note: this was originally merged as:
        https://review.opendev.org/#/c/750986/
    but the OctaviaBase references in templates was incorrect so the
    original patch was reverted through:
        https://review.opendev.org/#/c/763561/
    As the original patch had not been backported, it makes more sense to
    revert and get a correct fix in place and backport that instead of
    requiring backporting a series or backporting a squashed commit.

    Change-Id: Ib3476e53f89b50bae72b9c95a5d3dec51ed3de7e
    Related-Bug: #1863595
    (cherry picked from commit 6c11a949fc0b940ebc3815aeb86156bd60df5851)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/789405
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/866cc9bcba76631ded9acd30fd8ed897b40ead6d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 866cc9bcba76631ded9acd30fd8ed897b40ead6d
Author: Brent Eagles <email address hidden>
Date: Thu Sep 10 10:37:57 2020 -0230

    Create post-deploy.conf on update/upgrade for octavia

    Containers are restarted with new command lines during upgrade/updates
    before the external_deploy_task are run that create a configuration file
    that is used on the command line. This results in octavia services
    failing to start.

    Note: this was originally merged as:
        https://review.opendev.org/#/c/750986/
    but the OctaviaBase references in templates was incorrect so the
    original patch was reverted through:
        https://review.opendev.org/#/c/763561/
    As the original patch had not been backported, it makes more sense to
    revert and get a correct fix in place and backport that instead of
    requiring backporting a series or backporting a squashed commit.

    Conflicts:
      deployment/octavia/octavia-api-container-puppet.yaml

    Change-Id: Ib3476e53f89b50bae72b9c95a5d3dec51ed3de7e
    Related-Bug: #1863595
    (cherry picked from commit 6c11a949fc0b940ebc3815aeb86156bd60df5851)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common stein-eol release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/800511
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/27adf3bc7e6da4b12892956c48dd3a47b7470be8
Submitter: "Zuul (22348)"
Branch: master

commit 27adf3bc7e6da4b12892956c48dd3a47b7470be8
Author: Brent Eagles <email address hidden>
Date: Mon Jul 12 13:31:04 2021 -0230

    Ensure startup files exist before containers might be started

    Change Ib3476e53f89b50bae72b9c95a5d3dec51ed3de7e moved Octavia external
    deployment tasks to post deployment which occurs after the containers
    are launched. This causes them to fail on startup. While deployment
    continues, configuration is generated and the containers started later,
    it gives the appearance that the containers are failing at some point
    during deployment.

    This patch creates and empty file so the file passed in the command line
    exists when the container is first started. The caveat is that the
    configuration is incomplete until after the post deploy tasks have run.

    Related-bug: #1863595
    Change-Id: Id26edd4e7414c35cc0d2b4d743978a0d56c58fee