ovn agents are dead after ovn-controller update.

[Sofer Athlan-Guyot]

Hi,

after updating the ovn-controller the agent are deads:

| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+----------------------------+
| bc58a243-c9e1-48bc-aa25-23bc6a18e956 | OVN Controller agent | compute-0.redhat.local | | XXX | UP | ovn-controller |
| 32294417-48c4-507a-b7cd-5affaa62e009 | OVN Metadata agent | compute-0.redhat.local | | XXX | UP | neutron-ovn-metadata-agent |
| de3b2308-8cf1-4453-80a5-5bb7898c084f | OVN Controller Gateway agent | controller-0.redhat.local | | XXX | UP | ovn-controller |
| af5eb2d5-d51e-40f7-9dab-04d31588c034 | OVN Controller Gateway agent | controller-1.redhat.local | | XXX | UP | ovn-controller |
| 985726c7-09f8-45f7-bfe8-9e34cde5fe30 | OVN Controller Gateway agent | controller-2.redhat.local | | XXX | UP | ovn-controller |

with:

/var/log/containers/stdouts/ovn_controller.log:2023-02-20T18:17:00.231170529+00:00 stderr F 2023-02-20T18:17:00Z|00157|main|WARN|controller version - 22.12.1-20.27.0-70.6 mismatch with northd version - 22.03.2-20.21.0-61.4

Reported downstream there https://bugzilla.redhat.com/show_bug.cgi?id=2171896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874489

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874510

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874511

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874512

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/874492
Committed: https://opendev.org/openstack/puppet-tripleo/commit/178c9cb7f5f7d86c9154a1e13379e7a3002c3346
Submitter: "Zuul (22348)"
Branch: master

commit 178c9cb7f5f7d86c9154a1e13379e7a3002c3346
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 19:53:55 2023 +0100

    Do no set ovn-match-northd-version to true by default.

    This was introduced in commit ec21edf4d9d548c766dbe4ba9adfb045d0176bf4
    when ovn-controllers were not updated before northdb.

    Now, ovn-controller are updated first so the "Fail-safe upgrade"
    procedure[1] should not be used by default anymore.

    Closes-Bug: #2007870

    [1] https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html#fail-safe-upgrade

    Change-Id: I38829bc84ecc4cdeabd72e94db8341e86a2c1222

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/874485
Committed: https://opendev.org/openstack/puppet-tripleo/commit/c5959817dbcee177dfcc0f80681e44dcbb307a79
Submitter: "Zuul (22348)"
Branch: stable/zed

commit c5959817dbcee177dfcc0f80681e44dcbb307a79
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 19:53:55 2023 +0100

    Do no set ovn-match-northd-version to true by default.

    This was introduced in commit ec21edf4d9d548c766dbe4ba9adfb045d0176bf4
    when ovn-controllers were not updated before northdb.

    Now, ovn-controller are updated first so the "Fail-safe upgrade"
    procedure[1] should not be used by default anymore.

    Closes-Bug: #2007870

    [1] https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html#fail-safe-upgrade

    Change-Id: I38829bc84ecc4cdeabd72e94db8341e86a2c1222
    (cherry picked from commit 178c9cb7f5f7d86c9154a1e13379e7a3002c3346)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/874486
Committed: https://opendev.org/openstack/puppet-tripleo/commit/b1c87d04387b6af49e401cdc6234ee593f4a6ae8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b1c87d04387b6af49e401cdc6234ee593f4a6ae8
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 19:53:55 2023 +0100

    Do no set ovn-match-northd-version to true by default.

    This was introduced in commit ec21edf4d9d548c766dbe4ba9adfb045d0176bf4
    when ovn-controllers were not updated before northdb.

    Now, ovn-controller are updated first so the "Fail-safe upgrade"
    procedure[1] should not be used by default anymore.

    Closes-Bug: #2007870

    [1] https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html#fail-safe-upgrade

    Change-Id: I38829bc84ecc4cdeabd72e94db8341e86a2c1222
    (cherry picked from commit c5959817dbcee177dfcc0f80681e44dcbb307a79)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/874487
Committed: https://opendev.org/openstack/puppet-tripleo/commit/68b4a7c87d3af1200d18149365844d40683cb530
Submitter: "Zuul (22348)"
Branch: stable/train

commit 68b4a7c87d3af1200d18149365844d40683cb530
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 19:53:55 2023 +0100

    Do no set ovn-match-northd-version to true by default.

    This was introduced in commit ec21edf4d9d548c766dbe4ba9adfb045d0176bf4
    when ovn-controllers were not updated before northdb.

    Now, ovn-controller are updated first so the "Fail-safe upgrade"
    procedure[1] should not be used by default anymore.

    Closes-Bug: #2007870

    [1] https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html#fail-safe-upgrade

    Change-Id: I38829bc84ecc4cdeabd72e94db8341e86a2c1222
    (cherry picked from commit b1c87d04387b6af49e401cdc6234ee593f4a6ae8)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sofer Athlan-Guyot <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874489
Reason: See zed review instead: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874510

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874510
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/f835f5bc3975df27716dbe6d57d0a552efd17f74
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f835f5bc3975df27716dbe6d57d0a552efd17f74
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 20:22:21 2023 +0100

    Ensure ovn match northd parameter is set to false during update.

    As we are updating the ovn-controllers first that parameter shouldn't
    be set to true.

    Change-Id: I0e49e785232f5f97ab494ba81ac9b7cdb016f1b9
    Partial-Bug: #2007870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874512
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ffc14a0d300614e9adaaa607eaf35d4931f3313d
Submitter: "Zuul (22348)"
Branch: stable/train

commit ffc14a0d300614e9adaaa607eaf35d4931f3313d
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 20:22:21 2023 +0100

    Ensure ovn match northd parameter is set to false during update.

    As we are updating the ovn-controllers first that parameter shouldn't
    be set to true.

    Change-Id: I0e49e785232f5f97ab494ba81ac9b7cdb016f1b9
    Partial-Bug: #2007870
    (cherry picked from commit 6671d5028c53ad80091c0f41aa70bb1c2b70fcf3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874511
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/6671d5028c53ad80091c0f41aa70bb1c2b70fcf3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6671d5028c53ad80091c0f41aa70bb1c2b70fcf3
Author: Sofer Athlan-Guyot <email address hidden>
Date: Mon Feb 20 20:22:21 2023 +0100

    Ensure ovn match northd parameter is set to false during update.

    As we are updating the ovn-controllers first that parameter shouldn't
    be set to true.

    Change-Id: I0e49e785232f5f97ab494ba81ac9b7cdb016f1b9
    Partial-Bug: #2007870
    (cherry picked from commit f835f5bc3975df27716dbe6d57d0a552efd17f74)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo train-eol release.