potential performance issue when scheduling network segments

[Sahid Orentino]

During some investigations regarding segments we may have noticed performance issues related to the current algorithm that schedules network segments on hosts.

When an agent is reporting a change in segment, the process goes to the function `auto_schedule_new_network_segments` with the list of the segments that this host handles.

This function is retrieving from the segments the related networks, then we can notice that the algorithm is running a double for loop. That one iterates through network and per segments to schedule network segments on all hosts.

        for network_id in network_ids:
            for segment in segments:
                self._schedule_network(
                    payload.context, network_id, dhcp_notifier,
                        candidate_hosts=segment['hosts'])

Depending on the design chosen, in a setup that has hundred segments per host with hundred networks and potentially segments that share the same list of hosts, we will endup by calling _schedule_network 10000 times with duplication.

To avoid such duplication and unnecessary calls of _schedule_network for the same hosts we may want to provide a datastructure that is storing for each network the hosts already scheduled.

     for network_id in network_ids:
         for segment in segments:
             if not _already_scheduled(network_id, segment['hosts']):
                 self._schedule_network(
                     payload.context, network_id, dhcp_notifier,
                     candidate_hosts=segment['hosts'])

With this same scenario, and by using such algorithm we may reduce the number of call per the number of networks, 100.

Thanks,
s.

[Lajos Katona (lajos-katona)]

Thanks for the bug report, do you have perhaps time to implement this solution?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Sahid:

Good catch.

The segments belong to a single network only. You can use this field to execute or not the scheduling:

        for network_id in network_ids:
            for segment in (s for s in segments if s['network_id'] == network_id):
                self._schedule_network(
                    payload.context, network_id, dhcp_notifier,
                    candidate_hosts=segment['hosts'])

In each "network_id" loop, only those segments belonging to this network will be scheduled; each segment will be scheduled only once.

Thanks again for the bug report.

Regards.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/829022

[Sahid Orentino (sahid-ferdjaoui)]

With this patch we reduced the number off calls from about 1 000 000 to 1 000 that *only* for one neutron server.

This has real impact on infratructure.

This is also related to this bug https://bugs.launchpad.net/neutron/+bug/1952730 Where the proposed fix was to avoid this loop to happen but did not fixed the real root cause.

It's also related to that one, and should close that bug I guess.

https://bugs.launchpad.net/neutron/+bug/1951010

@lajos-katona, @rodolfo-alonso-hernandez, my feeling is that we should have this patch as backport potential and have it with highy priority.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/829022
Committed: https://opendev.org/openstack/neutron/commit/b5b519a4faa52677d5b9d5553217ed40aac70483
Submitter: "Zuul (22348)"
Branch: master

commit b5b519a4faa52677d5b9d5553217ed40aac70483
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Feb 14 13:52:10 2022 +0100

    segments: fix scheduling duplicate segments

    It has been reported by bug #1959750 that for a given network the
    agent is scheduling all segments reported by agent.

    This commit is fixing the issue by ensuring scheduling segments per
    network that they belong to only.
    A test has been updated to demonstract the fact that for a given
    network we only schedule the related segments.

    Closes-bug: #1959750
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    Change-Id: Icd57570004055903c6817a75d814ed65db3fa53c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/829956

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/829957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/829958

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/829956
Committed: https://opendev.org/openstack/neutron/commit/691496c365c6a8d6931b6a839b54ac74a741cd6a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 691496c365c6a8d6931b6a839b54ac74a741cd6a
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Feb 14 13:52:10 2022 +0100

    segments: fix scheduling duplicate segments

    It has been reported by bug #1959750 that for a given network the
    agent is scheduling all segments reported by agent.

    This commit is fixing the issue by ensuring scheduling segments per
    network that they belong to only.
    A test has been updated to demonstract the fact that for a given
    network we only schedule the related segments.

    Closes-bug: #1959750
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    Change-Id: Icd57570004055903c6817a75d814ed65db3fa53c
    (cherry picked from commit b5b519a4faa52677d5b9d5553217ed40aac70483)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/829958
Committed: https://opendev.org/openstack/neutron/commit/d06384867f693b0a656dac010906b085897c05ac
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d06384867f693b0a656dac010906b085897c05ac
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Feb 14 13:52:10 2022 +0100

    segments: fix scheduling duplicate segments

    It has been reported by bug #1959750 that for a given network the
    agent is scheduling all segments reported by agent.

    This commit is fixing the issue by ensuring scheduling segments per
    network that they belong to only.
    A test has been updated to demonstract the fact that for a given
    network we only schedule the related segments.

    Closes-bug: #1959750
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    Change-Id: Icd57570004055903c6817a75d814ed65db3fa53c
    (cherry picked from commit b5b519a4faa52677d5b9d5553217ed40aac70483)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/829957
Committed: https://opendev.org/openstack/neutron/commit/276f1541580c4b36b583d3d4bce76e2c44e00515
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 276f1541580c4b36b583d3d4bce76e2c44e00515
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Feb 14 13:52:10 2022 +0100

    segments: fix scheduling duplicate segments

    It has been reported by bug #1959750 that for a given network the
    agent is scheduling all segments reported by agent.

    This commit is fixing the issue by ensuring scheduling segments per
    network that they belong to only.
    A test has been updated to demonstract the fact that for a given
    network we only schedule the related segments.

    Closes-bug: #1959750
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    Change-Id: Icd57570004055903c6817a75d814ed65db3fa53c
    (cherry picked from commit b5b519a4faa52677d5b9d5553217ed40aac70483)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.2.0 release.