OVN metadata agent can be slow with large amount of subnets

[Miro Tomaska]

OVN metadata agent can take very long time (observed ~40s) to add cidrs under a metadata namespace tap interface when a network consist of many subnets (observed ~1700 subnets). The long processing time can result in ovn-metada-agent not having haproxy ready by the time the first VM cloud-init requests for its metadata. Thus resulting in VM missing metadata for proper operation.

Reproducing step:
- Create a network with hundreds or thousands of subnets under this network. The more subnets the more obvious the problem is
- Create a VM connected to the network from above. Make sure this is the first VM on the deployed compute node(hypervisor).
- Once VM is created, observe that VM's cloud-init request time out due to no response from 169.256.169.256/openstack
- Inspect ovn-metadata-agent log and notice this is due to ovn-metadata-agent taking very long time to process [1]

Possible solutions:
1. (Low hanging fruit?) See if there is a way to improve execution time of `ip.add` call. Perhaps passing a list of cidrs instead of a single cidr at the time can improve performance?
2. (more involved) refactor the code such that ovn-metadata-agent only adds a single cidr which belongs to the VM being created. Instead of unconditionally adding all cidrs for the network when the first VM is created(current implementation)

[1] https://github.com/openstack/neutron/blob/41bf8054017c72815226d5df50fd321b30fcba13/neutron/agent/ovn/metadata/agent.py#L488-L495

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/855677

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/861124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/855677
Committed: https://opendev.org/openstack/neutron/commit/81980146cbdcc42d2398f4e777e659fc59c3b55f
Submitter: "Zuul (22348)"
Branch: master

commit 81980146cbdcc42d2398f4e777e659fc59c3b55f
Author: Miro Tomaska <email address hidden>
Date: Fri Sep 2 10:26:11 2022 -0500

    Add and delete multiple ip addresses in one priv call

    Created new add_ip_addresses privileged function
    which takes an iterable of cidrs and adds them
    in one privileged call. This is so we dont have to
    take on additional priv overhead when calling
    add_ip_address in a loop.
    For parity, performed the same change on the
    delete_ip_address function.

    Closes-Bug: #1987281
    Partial-Bug: #1981113
    Change-Id: Ib1278af20c3b3b057712453cb249aba34b684a21

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/861124
Committed: https://opendev.org/openstack/neutron/commit/edf48e46a1f0227f84b05ab39da005393e5fa73f
Submitter: "Zuul (22348)"
Branch: master

commit edf48e46a1f0227f84b05ab39da005393e5fa73f
Author: Miro Tomaska <email address hidden>
Date: Wed Oct 12 08:42:18 2022 -0500

    Improve agent provision performance for large networks

    Before this patch, the metadata agent would provision network namespace
    for all subnets under a network(datapath) as soon as the first
    VM(vif port) was mounted on the chassis. This operation can take very
    long time for networks with lots of subnets. See the linked bug for
    more details.
    This patch changes this mechanism to "lazy load" where metadata agent
    provisions metadata namespace with only the subnets belonging to the
    active ports on the chassis. This results in virtually constant
    throughput not effected by the number of subnets.

    Closes-Bug: #1981113
    Change-Id: Ia2a66cfd3fd1380c5204109742d44f09160548d2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/869940

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/869941

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/869942

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/869943

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/869944

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/869945

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/870474

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/870504

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/869940
Reason: many backport conflicts outweight the benefit of backporting.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/869941
Reason: many backport conflicts outweight the benefit of backporting

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/869942
Reason: many backport conflicts outweight the benefit of backporting

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/869943
Reason: many backport conflicts outweight the benefit of backporting

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/869944
Reason: many backport conflicts outweight the benefit of backporting

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/869945
Reason: many backport conflicts outweight the benefit of backporting

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/870785

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/870786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/870787

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Miro Tomaska <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/870787
Reason: Decided to only backport to Wallaby. The potential backport problems outweight the benefit of this patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/870474
Committed: https://opendev.org/openstack/neutron/commit/bbaca0c86b3cb12aa3414a5e2921f86a8d01de7f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit bbaca0c86b3cb12aa3414a5e2921f86a8d01de7f
Author: Miro Tomaska <email address hidden>
Date: Wed Oct 12 08:42:18 2022 -0500

    Improve agent provision performance for large networks

    Before this patch, the metadata agent would provision network namespace
    for all subnets under a network(datapath) as soon as the first
    VM(vif port) was mounted on the chassis. This operation can take very
    long time for networks with lots of subnets. See the linked bug for
    more details.
    This patch changes this mechanism to "lazy load" where metadata agent
    provisions metadata namespace with only the subnets belonging to the
    active ports on the chassis. This results in virtually constant
    throughput not effected by the number of subnets.

    Merge Conflict:
            Implicitly removes now dead function
            ensure_all_networks_provisioned
            neutron/agent/ovn/metadata/agent.py

    Closes-Bug: #1981113
    Change-Id: Ia2a66cfd3fd1380c5204109742d44f09160548d2
    (cherry picked from commit edf3b3f191c2eae229a754dcbfc448fa41bd8bc3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/870504
Committed: https://opendev.org/openstack/neutron/commit/1d9ce0406883dc9ec386a01eba826af6fde6ceb1
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 1d9ce0406883dc9ec386a01eba826af6fde6ceb1
Author: Miro Tomaska <email address hidden>
Date: Wed Oct 12 08:42:18 2022 -0500

    Improve agent provision performance for large networks

    Before this patch, the metadata agent would provision network namespace
    for all subnets under a network(datapath) as soon as the first
    VM(vif port) was mounted on the chassis. This operation can take very
    long time for networks with lots of subnets. See the linked bug for
    more details.
    This patch changes this mechanism to "lazy load" where metadata agent
    provisions metadata namespace with only the subnets belonging to the
    active ports on the chassis. This results in virtually constant
    throughput not effected by the number of subnets.

    Merge Conflict:
            Using datapath_uuid :str in addition to net_name for
            teardown_datapath method to remain compatible with the
            method implementation in Yoga and before. Updated unit
            tests accordingly
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Closes-Bug: #1981113
    Change-Id: Ia2a66cfd3fd1380c5204109742d44f09160548d2
    (cherry picked from commit edf3b3f191c2eae229a754dcbfc448fa41bd8bc3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/870785
Committed: https://opendev.org/openstack/neutron/commit/44f95a48fc6346d009528245dac472f87131ce0a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 44f95a48fc6346d009528245dac472f87131ce0a
Author: Miro Tomaska <email address hidden>
Date: Wed Oct 12 08:42:18 2022 -0500

    Improve agent provision performance for large networks

    Before this patch, the metadata agent would provision network namespace
    for all subnets under a network(datapath) as soon as the first
    VM(vif port) was mounted on the chassis. This operation can take very
    long time for networks with lots of subnets. See the linked bug for
    more details.
    This patch changes this mechanism to "lazy load" where metadata agent
    provisions metadata namespace with only the subnets belonging to the
    active ports on the chassis. This results in virtually constant
    throughput not effected by the number of subnets.

    Merge Conflict:
            Using datapath_uuid :str in addition to net_name for
            teardown_datapath method to remain compatible with the
            method implementation in Yoga and before. Updated unit
            tests accordingly
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Closes-Bug: #1981113
    Change-Id: Ia2a66cfd3fd1380c5204109742d44f09160548d2
    (cherry picked from commit edf3b3f191c2eae229a754dcbfc448fa41bd8bc3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/870786
Committed: https://opendev.org/openstack/neutron/commit/9f90d18ad2bc5209c3caece51898401851ee738b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9f90d18ad2bc5209c3caece51898401851ee738b
Author: Miro Tomaska <email address hidden>
Date: Wed Oct 12 08:42:18 2022 -0500

    Improve agent provision performance for large networks

    Before this patch, the metadata agent would provision network namespace
    for all subnets under a network(datapath) as soon as the first
    VM(vif port) was mounted on the chassis. This operation can take very
    long time for networks with lots of subnets. See the linked bug for
    more details.
    This patch changes this mechanism to "lazy load" where metadata agent
    provisions metadata namespace with only the subnets belonging to the
    active ports on the chassis. This results in virtually constant
    throughput not effected by the number of subnets.

    Merge Conflict:
            Using datapath_uuid :str in addition to net_name for
            teardown_datapath method to remain compatible with the
            method implementation in Yoga and before. Updated unit
            tests accordingly
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Closes-Bug: #1981113
    Change-Id: Ia2a66cfd3fd1380c5204109742d44f09160548d2
    (cherry picked from commit edf3b3f191c2eae229a754dcbfc448fa41bd8bc3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.