Metadata service broken after minor neutron update when OVN 21.09+ is used

[Ihar Hrachyshka]

Originally reported at: https://bugzilla.redhat.com/show_bug.cgi?id=2093901

Prerequisites:

1. OVN 21.09+ that includes https://github.com/ovn-org/ovn/commit/3ae8470edc648b7401433a22a9f15053cc7e666d
2. Existing metadata namespace created by OVN agent before commit https://review.opendev.org/c/openstack/neutron/+/768462

Steps to reproduce:
1. Neutron OVN metadata agent updated to include the patch from prereq (2).
2. Neutron OVN metadata agent is restarted. It will create a new network namespace to host the metadata vif. It will also remove the old vif.
3. curl http://169.254.169.254/latest/meta-data/ from a VM that is hosted on the same node. It fails.

This happens because the agent first creates new vif, then deletes the old vif. Which puts OVN into a situation where 2 interfaces exist in parallel assigned to the same LSP. This scenario is considered invalid by OVN core team. There's a patch up for review for OVN core to handle the situation more gracefully: https://patchwork<email address hidden>/ This patch will not leave metadata service broken, but it will trigger full recompute in OVN. So we should not rely on its mechanics. Instead Neutron should make sure that no two vifs carry the same iface-id at the same time.

The reason why this was not a problem with OVN 21.06 or earlier is because the patch referred in prereq (1) changed the behavior in this invalid / undefined scenario.

[Ihar Hrachyshka (ihar-hrachyshka)]

Fix: https://review.opendev.org/c/openstack/neutron/+/864777

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/864977

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/864978

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/864979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/864980

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/865064

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/864981

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864777
Committed: https://opendev.org/openstack/neutron/commit/3093aaab13dd6ba04ef0e686eb4c6cc386c58941
Submitter: "Zuul (22348)"
Branch: master

commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ihar Hrachyshka <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/864981
Reason: The patch that triggered the failing scenario - https://review.opendev.org/c/openstack/neutron/+/768462 - was not backported to this branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ihar Hrachyshka <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/865064
Reason: The patch that triggered the failing scenario - https://review.opendev.org/c/openstack/neutron/+/768462 - was not backported to this branch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864979
Committed: https://opendev.org/openstack/neutron/commit/c5a3ea4a1027f72f3f6c5f3304a96b900d8eba95
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c5a3ea4a1027f72f3f6c5f3304a96b900d8eba95
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865064
Committed: https://opendev.org/openstack/neutron/commit/e6878c12c3064a5a13589dd4df05a053e3f96bee
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit e6878c12c3064a5a13589dd4df05a053e3f96bee
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Conflicts:
          neutron/agent/ovn/metadata/agent.py

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864977
Committed: https://opendev.org/openstack/neutron/commit/fcaff8c80db23ee92756b48579bf898c88d7b932
Submitter: "Zuul (22348)"
Branch: stable/zed

commit fcaff8c80db23ee92756b48579bf898c88d7b932
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864980
Committed: https://opendev.org/openstack/neutron/commit/37ced67abafa150aa446fa06583983d81e499b31
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 37ced67abafa150aa446fa06583983d81e499b31
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864981
Committed: https://opendev.org/openstack/neutron/commit/4da5638baea22db869c2ef8e1bf5bfc40cdbfdfc
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 4da5638baea22db869c2ef8e1bf5bfc40cdbfdfc
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Conflicts:
          neutron/agent/ovn/metadata/agent.py

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/864978
Committed: https://opendev.org/openstack/neutron/commit/e62c81a5704316ac1b42ce546e31f1732fa9b986
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit e62c81a5704316ac1b42ce546e31f1732fa9b986
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Nov 16 18:47:04 2022 +0000

    ovn: first tear down old metadata namespaces, then deploy new

    While the reverse order may work, it's considered invalid by OVN and not
    guaranteed to work properly since OVN may not necessarily know which of
    two ports is the one to configure.

    This configuration also triggered a bug in OVN where tearing down a port
    after deploying a new one resulted in removing flows that serve the
    port.

    There is a patch up for review for OVN [1] to better handle multiple
    assignment of the same port, but it doesn't make the setup any more
    valid.

    [1] http://patchwork<email address hidden>/

    Closes-Bug: #1997092
    Change-Id: Ic7dbc4e8b00423e58f69646a9e3cedc6f72d6c63
    (cherry picked from commit 3093aaab13dd6ba04ef0e686eb4c6cc386c58941)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.