project/instances/attach_interface has O(N) scaling time complexity for opening form

[Nicolas Bock]

[ Impact ]

The time complexity of opening the project/instances/attach_interface form box is O(N) where N is the number of networks in the project with a large prefactor.

This is due to

https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/dashboards/project/instances/utils.py#L210

Which loops over the networks and requests the ports associated with the network. For large projects this scaling behavior can become prohibitive.

The patch [1] addresses this issue by reducing the number of API calls and hence the prefactor of the algorithm.

[ Test Plan ]

In order to reproduce the issue, create a Nova VM and then add many networks. On the instances tab in the Horizon UI click on "attach interface" for the VM. It will take a moment for the dialog to appear. The exact time until the dialog appears will depend on the number of networks linearly.

With [1] the time it takes for the dialog box to appear will be significantly shorter.

[ Where problems could occur ]

The patch [1] affects the "attach interface" dialog box and could break this UI feature in case something was wrong with the implementation. It is also possible that due to a bug in the implementation some networks are missing from the dialog.

[1] https://review.opendev.org/c/openstack/horizon/+/866895

[Nicolas Bock (nicolasbock)]

https://review.opendev.org/c/openstack/horizon/+/809230/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/horizon/+/830936

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/809230
Committed: https://opendev.org/openstack/horizon/commit/9f5d659d160d24d359992dda22fecb0db6b6f805
Submitter: "Zuul (22348)"
Branch: master

commit 9f5d659d160d24d359992dda22fecb0db6b6f805
Author: Nicolas Bock <email address hidden>
Date: Wed Sep 15 09:39:31 2021 -0600

    Get ports directly instead of via loop

    In order to get a list of available ports, the current implementation
    of the form code in `project/instances/attach_interface` loops through
    the available networks in the project and requests a list of ports for
    each via the Neutron API. This implementation is O(N) in time with a
    large prefactor (the time for a Neutron API call) where N is the
    number of networks.

    Instead of calling the Neutron API for each network this change uses
    one call to the Neutron API to get the list of ports on all networks
    and filters this list via a list comprehension. While this
    implementation is still O(N) the prefactor is significantly smaller.

    Closes-Bug: #1943639
    Change-Id: I8fd32c3aad22d8ef7f57201f5144f6b2e357ef10
    Signed-off-by: Nicolas Bock <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/horizon/+/831035

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 22.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/830936
Committed: https://opendev.org/openstack/horizon/commit/516e57bc895c184756900fa9c6e15110e7ecf805
Submitter: "Zuul (22348)"
Branch: master

commit 516e57bc895c184756900fa9c6e15110e7ecf805
Author: Akihiro Motoki <email address hidden>
Date: Fri Feb 25 15:31:41 2022 +0900

    Add UT coverage for attach_interface by port

    Previously attach_interface tests in project/instances/tests.py
    did not cover the case of attach_interface by port.
    This commit adds UT for such cases.

    Details:
    - The second call of the mocked network_list_for_tenant retured
      an empty list. It sounds tricky to change the return value for
      network_list_for_tenant() in two calls, but this trick was used
      to skip the processing of port_field_data.
      This should return a same value for the two calls to test
      port_field_data() function.
    - To test the behavior of attach_interface by "port",
      an unbound port (whose device_owner/device_id of the port is empty)
      is required, so this commit adds it to neutron_data.py.
    - test_interface_attach_get() covers a list of choices when "By Port"
      is selected in the form.
    - test_interface_attach_post_by_port() is added.

    Related to the addition of an unbound port to neutron_data.py,
    the following other tests are adjusted.
    They assumed that all non-network ports are owned by servers,
    but it is no longer true as an unbound port is added to the test data.
    Note that associating an unbound port with a floating IP is a valid
    operation in neutron, so there is no problem to adjust UTs.

    - openstack_dashboard/dashboards/project/floating_ips/tests.py
    - openstack_dashboard/test/unit/api/test_neutron.py

    Related-Bug: #1943639
    Change-Id: Ib0ee342463e5668858078db43c04fe0a1be6e995

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/831035
Committed: https://opendev.org/openstack/horizon/commit/1561fcce55371c6b2d93a31b83ca533dea82f746
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 1561fcce55371c6b2d93a31b83ca533dea82f746
Author: Nicolas Bock <email address hidden>
Date: Wed Sep 15 09:39:31 2021 -0600

    Get ports directly instead of via loop

    In order to get a list of available ports, the current implementation
    of the form code in `project/instances/attach_interface` loops through
    the available networks in the project and requests a list of ports for
    each via the Neutron API. This implementation is O(N) in time with a
    large prefactor (the time for a Neutron API call) where N is the
    number of networks.

    Instead of calling the Neutron API for each network this change uses
    one call to the Neutron API to get the list of ports on all networks
    and filters this list via a list comprehension. While this
    implementation is still O(N) the prefactor is significantly smaller.

    Includes some unit test fixes to workaround the fact that patch
    6ac31e0 is not backported beyond Y.

    Closes-Bug: #1943639
    Change-Id: I8fd32c3aad22d8ef7f57201f5144f6b2e357ef10
    Signed-off-by: Nicolas Bock <email address hidden>
    (cherry picked from commit 9f5d659d160d24d359992dda22fecb0db6b6f805)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/horizon/+/866891

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/horizon/+/866894

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/horizon/+/866895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/horizon/+/867150

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 20.1.3 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/866891
Committed: https://opendev.org/openstack/horizon/commit/c5a53bc63b269209285ad623275b3f545e509b8f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c5a53bc63b269209285ad623275b3f545e509b8f
Author: Nicolas Bock <email address hidden>
Date: Wed Sep 15 09:39:31 2021 -0600

    Get ports directly instead of via loop

    In order to get a list of available ports, the current implementation
    of the form code in `project/instances/attach_interface` loops through
    the available networks in the project and requests a list of ports for
    each via the Neutron API. This implementation is O(N) in time with a
    large prefactor (the time for a Neutron API call) where N is the
    number of networks.

    Instead of calling the Neutron API for each network this change uses
    one call to the Neutron API to get the list of ports on all networks
    and filters this list via a list comprehension. While this
    implementation is still O(N) the prefactor is significantly smaller.

    Includes some unit test fixes to workaround the fact that patch
    6ac31e0 is not backported beyond Y.

    Closes-Bug: #1943639
    Change-Id: I8fd32c3aad22d8ef7f57201f5144f6b2e357ef10
    Signed-off-by: Nicolas Bock <email address hidden>
    (cherry picked from commit 9f5d659d160d24d359992dda22fecb0db6b6f805)
    (cherry picked from commit 1561fcce55371c6b2d93a31b83ca533dea82f746)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/866895
Committed: https://opendev.org/openstack/horizon/commit/ec39137c2d23b0da3bdc7a7ffcd0997a18f4c5ca
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ec39137c2d23b0da3bdc7a7ffcd0997a18f4c5ca
Author: Nicolas Bock <email address hidden>
Date: Wed Sep 15 09:39:31 2021 -0600

    Get ports directly instead of via loop

    In order to get a list of available ports, the current implementation
    of the form code in `project/instances/attach_interface` loops through
    the available networks in the project and requests a list of ports for
    each via the Neutron API. This implementation is O(N) in time with a
    large prefactor (the time for a Neutron API call) where N is the
    number of networks.

    Instead of calling the Neutron API for each network this change uses
    one call to the Neutron API to get the list of ports on all networks
    and filters this list via a list comprehension. While this
    implementation is still O(N) the prefactor is significantly smaller.

    Includes some unit test fixes to workaround the fact that patch
    6ac31e0 is not backported beyond Y.

    Closes-Bug: #1943639
    Change-Id: I8fd32c3aad22d8ef7f57201f5144f6b2e357ef10
    Signed-off-by: Nicolas Bock <email address hidden>
    (cherry picked from commit 9f5d659d160d24d359992dda22fecb0db6b6f805)
    (cherry picked from commit 1561fcce55371c6b2d93a31b83ca533dea82f746)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "focal-ussuri.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Simon Quigley (tsimonq2)]

Please add both patches to debian/patches/series.

[Steve Langasek (vorlon)]

SRUs of the Openstack packages are typically managed by the Canonical Openstack team, so I believe sponsorship through the ubuntu-sponsors team is not necessary and am unsubscribing. If you do need sponsorship from this team, please re-subscribe us.

[Edward Hope-Morley (hopem)]

This patch was merged to master during the Yoga development cycle and is available in the following point releases upstream:

  (yoga) 22.0.0
  (xena) 20.1.3

The Ubuntu archives currently have the following versions:

  Yoga - 4:22.1.0-0ubuntu2.1~cloud0
  Xena - 4:20.1.4-0ubuntu1~cloud1
  Wallaby - 4:19.4.0-0ubuntu1~cloud1
  Victoria - 4:18.6.4-0ubuntu1~cloud1
  Ussuri - 3:18.3.5-0ubuntu2.1

So to get this patch backported to Ussuri, it first needs to be SRUd to Wallaby and Victoria UCA.

[Nicolas Bock (nicolasbock)]

Simon, do you mean the "Add UT coverage for attach_interface by port" patch when you say "both" patches? That patch is not in Ussuri for example. I will have to backport it first.

[Corey Bryant (corey.bryant)]

Nicolas, thanks for your work on this. New package versions have been uploaded to wallaby-staging, victoria-staging and the focal unapproved queue.

[Corey Bryant (corey.bryant)]

Hello Nicolas, or anyone else affected,

Accepted horizon into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Nicolas, or anyone else affected,

Accepted horizon into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Timo Aaltonen (tjaalton)]

Hello Nicolas, or anyone else affected,

Accepted horizon into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/horizon/3:18.3.5-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Nicolas, or anyone else affected,

Accepted horizon into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Nicolas Bock (nicolasbock)]

Verified focal:

- deploy Focal / Ussuri (openstack-dashboard-18.3.5-0ubuntu2.1)
- create a server
- create 200 networks
- click on 'attach interface' in dashboard UI for server instance
- install openstack-dashboard-18.3.5-0ubuntu2.2
- click on 'attach interface' in dashboard UI for server instance
- confirmed that the UI dialog appears significantly faster

[Nicolas Bock (nicolasbock)]

Verified bionic-ussuri:

- deploy Bionic / Ussuri (openstack-dashboard-18.3.5-0ubuntu2.1~cloud0)
- create a server
- create 200 networks
- click on 'attach interface' in dashboard UI for server instance
- install openstack-dashboard-18.3.5-0ubuntu2.2~cloud0
- click on 'attach interface' in dashboard UI for server instance
- confirmed that the UI dialog appears significantly faster

[Nicolas Bock (nicolasbock)]

Verified focal-victoria:

- deploy Focal / Victoria (openstack-dashboard-18.6.4-0ubuntu1~cloud1)
- create a server
- create 200 networks
- click on 'attach interface' in dashboard UI for server instance
- install openstack-dashboard-18.6.4-0ubuntu1~cloud2
- click on 'attach interface' in dashboard UI for server instance
- confirmed that the UI dialog appears significantly faster

[Nicolas Bock (nicolasbock)]

Verified focal-wallaby:

- deploy Focal / Wallaby (openstack-dashboard-19.4.0-0ubuntu1~cloud1)
- create a server
- create 200 networks
- click on 'attach interface' in dashboard UI for server instance
- install openstack-dashboard-19.4.0-0ubuntu1~cloud2
- click on 'attach interface' in dashboard UI for server instance
- confirmed that the UI dialog appears significantly faster

[Launchpad Janitor (janitor)]

This bug was fixed in the package horizon - 3:18.3.5-0ubuntu2.2

---------------
horizon (3:18.3.5-0ubuntu2.2) focal; urgency=medium

  * d/p/get-ports-directly.patch: Get ports directly instead of via
    loop (LP: #1943639).

 -- Nicolas Bock <email address hidden> Tue, 22 Aug 2023 20:58:11 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for horizon has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 4:19.4.0-0ubuntu1~cloud2
---------------

 horizon (4:19.4.0-0ubuntu1~cloud2) focal-wallaby; urgency=medium
 .
   * d/p/get-ports-directly.patch: Get ports directly instead of via
     loop (LP: #1943639).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 4:18.6.4-0ubuntu1~cloud2
---------------

 horizon (4:18.6.4-0ubuntu1~cloud2) focal-victoria; urgency=medium
 .
   * d/p/get-ports-directly.patch: Get ports directly instead of via
     loop (LP: #1943639).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 3:18.3.5-0ubuntu2.2~cloud0
---------------

 horizon (3:18.3.5-0ubuntu2.2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 horizon (3:18.3.5-0ubuntu2.2) focal; urgency=medium
 .
   * d/p/get-ports-directly.patch: Get ports directly instead of via
     loop (LP: #1943639).

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Nicolas Bock <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/horizon/+/867150

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon wallaby-eom release.