Creating a trunk with many subports is really slow

[Daniel Alvarez]

When running rally create and list trunk scenario [0], the create_trunk operation with 125 subports takes almost 180 seconds to complete. The reason why this happens is that we're updating the binding information for every subport [1]

If we could instead bulk update all the subports with a single call to the plugin, the operation would be much faster. On the other hand but this is not yet supported in ML2 plugin AFAIK.

[0] https://opendev.org/openstack/rally-openstack/src/tag/1.5.0/rally_openstack/scenarios/neutron/trunk.py#L39
[1] https://github.com/openstack/networking-ovn/blob/6.0.0.0rc1/networking_ovn/ml2/trunk_driver.py#L59

[Nate Johnston (nate-johnston)]

Is _set_sub_ports being called once with information about all the subports, or is it being called from the ML2 level once per subport?

[Daniel Alvarez (dalvarezs)]

@Nate: _set_sub_ports is called from the OVN Trunk driver [0] upon trunk creation or when adding subports to an existing trunk. The trunk driver will then call the ML2 plugin to update each subport one by one. Let me know if it doesn't make sense, please.

[0] https://github.com/openstack/networking-ovn/blob/6.0.0.0rc1/networking_ovn/ml2/trunk_driver.py#L68

[Daniel Alvarez (dalvarezs)]

Talking to Slaweq looks like one -ugly- possibility could be to update the DB directly from networking-ovn trunk driver to avoid calling port_update on the plugin which seems to be *really* slow.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676182
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=41f6d622abc65ca4c10b08f8c022915df2809255
Submitter: Zuul
Branch: master

commit 41f6d622abc65ca4c10b08f8c022915df2809255
Author: Maciej Józefczyk <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 19.7 for 95%ile
    from 99 sec to 14.9 for 50%ile

    The same thing has been done for Trunk deletion.

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Closes-Bug: #1834637

    Change-Id: I020ba64618b2eb76c627632b0575896e88d1fcf8

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/682596

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682599

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/682605

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/682606

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/682633

[OpenStack Infra (hudson-openstack)]

Change abandoned by Daniel Alvarez (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/682596
Reason: in favor of https://review.opendev.org/#/c/682633/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682633
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=9b71675a0ed1df09a9be0dfeab520490ebdd52d3
Submitter: Zuul
Branch: stable/stein

commit 9b71675a0ed1df09a9be0dfeab520490ebdd52d3
Author: Daniel Alvarez <email address hidden>
Date: Tue Sep 17 11:52:21 2019 +0200

    use trunk constants from neutron-lib

    The trunk constants live in neutron-lib now. This patch switches over
    to them in prep for https://review.openstack.org/#/c/649672/

    Related-Bug: #1834637

    Conflicts:
        networking_ovn/ml2/trunk_driver.py
        networking_ovn/tests/functional/test_trunk_driver.py
        networking_ovn/tests/unit/ml2/test_trunk_driver.py

    Change-Id: Ia5739ca39998a842df51d6ccd68717e842895c4d
    (cherry-picked from commit 814774a45791fd67ae7175165d5bf10f13989595)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682599
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1d58b0e680ee3da65a58620639b220bd972318fe
Submitter: Zuul
Branch: stable/stein

commit 1d58b0e680ee3da65a58620639b220bd972318fe
Author: Maciej Józefczyk <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 19.7 for 95%ile
    from 99 sec to 14.9 for 50%ile

    The same thing has been done for Trunk deletion.

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Closes-Bug: #1834637

    Conflicts:
        networking_ovn/ml2/trunk_driver.py
        networking_ovn/tests/unit/ml2/test_trunk_driver.py

    Change-Id: I020ba64618b2eb76c627632b0575896e88d1fcf8
    (cherry picked from commit 41f6d622abc65ca4c10b08f8c022915df2809255)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 7.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682605
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1e2e424f5d90be32edaf701d1871701bf4b5d522
Submitter: Zuul
Branch: stable/rocky

commit 1e2e424f5d90be32edaf701d1871701bf4b5d522
Author: Maciej Józefczyk <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 19.7 for 95%ile
    from 99 sec to 14.9 for 50%ile

    The same thing has been done for Trunk deletion.

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Closes-Bug: #1834637

    Conflicts:
        networking_ovn/ml2/trunk_driver.py
        networking_ovn/tests/unit/ml2/test_trunk_driver.py

    Change-Id: I020ba64618b2eb76c627632b0575896e88d1fcf8
    (cherry picked from commit 843200b9ce710d6973c0308b1b976a065efd6ce3)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/685313

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/685315

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.opendev.org/685313
Reason: This was supposed to go to stable/train

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685206
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=2e0832f7b8bfc31780b657aa0abda4e8b244fbbd
Submitter: Zuul
Branch: master

commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:46:40 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 41f6d622abc65ca4c10b08f8c022915df2809255.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I63acf0c809aa158aeba1c79c7c9083966ffd58a3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685315
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=60ca3c7809cba359871ed7220715a637448d9362
Submitter: Zuul
Branch: stable/train

commit 60ca3c7809cba359871ed7220715a637448d9362
Author: Jakub Libosvar <email address hidden>
Date: Fri Sep 27 12:23:09 2019 +0200

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 41f6d622abc65ca4c10b08f8c022915df2809255.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I244b937e916474703abf0a0c802e13bcd5a54918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685209
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4f2301a060ea8ce941b6a9c48d731537ce32494a
Submitter: Zuul
Branch: stable/stein

commit 4f2301a060ea8ce941b6a9c48d731537ce32494a
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:50:19 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 1d58b0e680ee3da65a58620639b220bd972318fe.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I55d224faead40da99b9c9f8ded5117a9b0bc288c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685207
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=3205f887fcc9a509c1a281f2d6d3788ff5147ba4
Submitter: Zuul
Branch: stable/rocky

commit 3205f887fcc9a509c1a281f2d6d3788ff5147ba4
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:47:10 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 1e2e424f5d90be32edaf701d1871701bf4b5d522.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I862751fabcd590e91d0ded00dd1cfeb9e6753f2f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686571

[Flavio Fernandes (ffernand)]

I had these loops running over the weekend [1] and was unable to reproduce the bug after applying the 686571 changes in patchset 2.

As you can see in the output below [2], I did see some failures in test, but they were all related to nova unable to ssh into a vm [3] after creation; which I consider outside the scope of bug 1834637.

[1]:

# main shell session
while : ; do \
   cd /opt/stack/tempest && while [ $? -eq 0 ]; \
   do sleep 1 ; echo '' ; date +"%D %T %Z" | tee -a /vagrant/tempest.log ; \
   tempest run --regex \
      neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle 2>&1 | tee -a /vagrant/tempest.log ; \
   done ; \
   sleep 30 ; \
done

# on another session
while : ; do openstack port list ; sleep 1 ; done

# on another session
while : ; do openstack port create --network private blablabla ; \
sleep 3 ; openstack port delete blablabla && echo deleted ; sleep 3 ; done

[2]:

$ ls -lah tempest.log
-rw-r--r-- 1 flaviof flaviof 3.4M Oct 7 04:59 tempest.log
$ grep -c "... ok" tempest.log
2340
$ grep -c "... FAILED" tempest.log
27

[3]:

$ grep -c 'File "tempest/lib/common/ssh.py", line 107, in _get_ssh_connection' tempest.log
54 <== 2 failed attempts for each one of the 27 failed tests

See attached log for more details.

[Flavio Fernandes (ffernand)]

Attaching tempest.log.before_fix, which shows how it is really needed to have context
instantiated as part of transaction.

$ grep -c "... ok" tempest.log.before_fix
84

$ grep -c "... FAILED" tempest.log.before_fix
2

$ grep -nA12 "... FAILED" tempest.log.before_fix
472:{0} neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle [236280.466269s] ... FAILED
473-
474-Captured traceback:
475-~~~~~~~~~~~~~~~~~~~
476- Traceback (most recent call last):
477- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 201, in test_trunk_subport_lifecycle
478- self.wait_for_server_active(server=vm.server)
479- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/base.py", line 380, in wait_for_server_active
480- server, constants.SERVER_STATUS_ACTIVE, client)
481- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/base.py", line 370, in wait_for_server_status
482- waiters.wait_for_server_status(client, server['id'], status, **kwargs)
483- File "tempest/common/waiters.py", line 76, in wait_for_server_status
484- server_id=server_id)
--
778:{0} neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle [102.238741s] ... FAILED
779-
780-Captured traceback:
781-~~~~~~~~~~~~~~~~~~~
782- Traceback (most recent call last):
783- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 230, in test_trunk_subport_lifecycle
784- self._wait_for_port(port, status=constants.DOWN)
785- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 141, in _wait_for_port
786- "status {!r}.".format(port['id'], status)))
787- File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 82, in wait_until_true
788- raise exception
789- RuntimeError: Timed out waiting for port u'40509745-1509-4077-9cce-b0bf86070084' to transition to get status 'DOWN'.
790-

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686571
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=7bdf2eb824083b6785d78dd67b3effe071e3f7a4
Submitter: Zuul
Branch: master

commit 7bdf2eb824083b6785d78dd67b3effe071e3f7a4
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/687919

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/687930

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687971

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/687972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687919
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1e20b9bf866e3001ad1cd71186caada647c54929
Submitter: Zuul
Branch: stable/train

commit 1e20b9bf866e3001ad1cd71186caada647c54929
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    ---
    Also, merge squashed:

    Enable ovs-vswitchd and ovsdb-server for rally

    The ovs-vswitchd and ovsdb-server services were missing in the rally
    definition. Usually the jobs inherts from our base class which enables
    them but rally has a different job parent.

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    Co-authored-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 7bdf2eb824083b6785d78dd67b3effe071e3f7a4)
    (cherry picked from commit 202e804c64c9c6d7160496987a4b13bd9f9ee6b6)
    (cherry picked from commit 7e725331dbaf5845d6a45fa1cf8035d6a61ac744)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lucas Alvares Gomes (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/682606

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687930
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=c6dee38266b662a2bb90c95af891045c731daf41
Submitter: Zuul
Branch: stable/stein

commit c6dee38266b662a2bb90c95af891045c731daf41
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit 1e20b9bf866e3001ad1cd71186caada647c54929)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687972
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=01d86c3c97b196279769522532ccdef2c3bf52ec
Submitter: Zuul
Branch: stable/queens

commit 01d86c3c97b196279769522532ccdef2c3bf52ec
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Note: While this is a back-merge from Rocky, special care was
    needed to account for the fact that in Queens release there
    is only one port binding for a db_port. Thus no iteration over
    db_port.bindings is used.

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit 62eb828186f9248e99fa12cc9c0c8397e1520514)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687971
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=62eb828186f9248e99fa12cc9c0c8397e1520514
Submitter: Zuul
Branch: stable/rocky

commit 62eb828186f9248e99fa12cc9c0c8397e1520514
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit c6dee38266b662a2bb90c95af891045c731daf41)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 4.0.4 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/692610

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/692619

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/692620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/692621

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/693108

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: master
Review: https://review.opendev.org/693107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693268

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: master
Review: https://review.opendev.org/693268

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/692610

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/693108

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/692621

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/692620

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: master
Review: https://review.opendev.org/691892

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/692619

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/693270

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/695760

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/698863

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/698873

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/698877

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Fernandes (<email address hidden>) on branch: master
Review: https://review.opendev.org/695760
Reason: This is now: https://review.opendev.org/#/c/701646/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698863
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=c418dd720b9be9047d779f32407c474e90cb0002
Submitter: Zuul
Branch: stable/train

commit c418dd720b9be9047d779f32407c474e90cb0002
Author: Flavio Fernandes <email address hidden>
Date: Fri Nov 22 15:11:48 2019 -0500

    Set binding profile directly from OVNTrunkDriver (redo cont.)

    This is a tweak to changes done to fix bug 1834637. Specifically,
    this change addresses scaling. The previous gerrit change had
    modifications to all OVN sub-ports performed as a single
    transaction. That did not account for race-condition on neutron
    DB queries, which leads to timeouts under heavy loads.

    Another cleanup done by this change is to fold an additional
    update on neutron db into ovn trunk driver. That saves
    update from doing another database transaction, thus making
    it faster. The no longer needed function in mech_driver was
    called _update_subport_host_if_needed

    By breaking the iteration into multiple transactions, the
    change in time is marginal:

      Service-level agreement
      NeutronTrunks :: neutron.create_trunk

      from 34.2 sec to 35.6 for 50%ile
      from 35.6 sec to 36.1 for 95%ile

    This patch doesn't go to master as it's frozen now because of
    the code being merged to Neutron. The patch will be sent to master
    Neutron once code migration is finished.

    Change-Id: I7de3ac2a7cf8869ead8ab5fbb34a9861a96d3a0c
    Closes-Bug: #1834637
    Co-authored-by: Maciej Józefczyk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698873
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=e669382b251b5fec28dba8e40a03c7b8f23f51d3
Submitter: Zuul
Branch: stable/stein

commit e669382b251b5fec28dba8e40a03c7b8f23f51d3
Author: Flavio Fernandes <email address hidden>
Date: Fri Nov 22 15:11:48 2019 -0500

    Set binding profile directly from OVNTrunkDriver (redo cont.)

    This is a tweak to changes done to fix bug 1834637. Specifically,
    this change addresses scaling. The previous gerrit change had
    modifications to all OVN sub-ports performed as a single
    transaction. That did not account for race-condition on neutron
    DB queries, which leads to timeouts under heavy loads.

    Another cleanup done by this change is to fold an additional
    update on neutron db into ovn trunk driver. That saves
    update from doing another database transaction, thus making
    it faster. The no longer needed function in mech_driver was
    called _update_subport_host_if_needed

    By breaking the iteration into multiple transactions, the
    change in time is marginal:

      Service-level agreement
      NeutronTrunks :: neutron.create_trunk

      from 34.2 sec to 35.6 for 50%ile
      from 35.6 sec to 36.1 for 95%ile

    This patch doesn't go to master networking-ovn. It
    has been migrated to master Neutron [1].

    [1]: https://review.opendev.org/#/c/701646/

    Change-Id: I7de3ac2a7cf8869ead8ab5fbb34a9861a96d3a0c
    Closes-Bug: #1834637
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit c418dd720b9be9047d779f32407c474e90cb0002)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698877
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=7f03767fd2e1eb669e28a3261b253e3aa3f37486
Submitter: Zuul
Branch: stable/rocky

commit 7f03767fd2e1eb669e28a3261b253e3aa3f37486
Author: Flavio Fernandes <email address hidden>
Date: Fri Nov 22 15:11:48 2019 -0500

    Set binding profile directly from OVNTrunkDriver (redo cont.)

    This is a tweak to changes done to fix bug 1834637. Specifically,
    this change addresses scaling. The previous gerrit change had
    modifications to all OVN sub-ports performed as a single
    transaction. That did not account for race-condition on neutron
    DB queries, which leads to timeouts under heavy loads.

    Another cleanup done by this change is to fold an additional
    update on neutron db into ovn trunk driver. That saves
    update from doing another database transaction, thus making
    it faster. The no longer needed function in mech_driver was
    called _update_subport_host_if_needed

    By breaking the iteration into multiple transactions, the
    change in time is marginal:

      Service-level agreement
      NeutronTrunks :: neutron.create_trunk

      from 34.2 sec to 35.6 for 50%ile
      from 35.6 sec to 36.1 for 95%ile

    This patch doesn't go to master networking-ovn. It
    has been migrated to master Neutron [1].

    [1]: https://review.opendev.org/#/c/701646/

    Change-Id: I7de3ac2a7cf8869ead8ab5fbb34a9861a96d3a0c
    Closes-Bug: #1834637
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit c418dd720b9be9047d779f32407c474e90cb0002)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693270
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4bed17daa762f9787391484945a445eabd617c04
Submitter: Zuul
Branch: stable/queens

commit 4bed17daa762f9787391484945a445eabd617c04
Author: Flavio Fernandes <email address hidden>
Date: Fri Nov 1 16:33:12 2019 -0400

    Set binding profile directly from OVNTrunkDriver (redo cont.)

    This is a tweak to changes done to fix bug 1834637. Specifically,
    this change addresses scaling. The previous gerrit change had
    modifications to all OVN sub-ports performed as a single
    transaction. That did not account for race-condition on neutron
    DB queries, which leads to timeouts under heavy loads.

    Another cleanup done by this change is to fold an additional
    update on neutron db into ovn trunk driver. That saves
    update from doing another database transaction, thus making
    it faster. The no longer needed function in mech_driver was
    called _update_subport_host_if_needed

    By breaking the iteration into multiple transactions, the
    change in time is marginal:

      Service-level agreement
      NeutronTrunks :: neutron.create_trunk

      from 34.2 sec to 35.6 for 50%ile
      from 35.6 sec to 36.1 for 95%ile

    This patch doesn't go to master networking-ovn. It
    has been migrated to master Neutron [1].

    [1]: https://review.opendev.org/#/c/701646/

    Depends-On: https://review.opendev.org/#/c/695693/
    Change-Id: I7de3ac2a7cf8869ead8ab5fbb34a9861a96d3a0c
    Closes-Bug: #1834637
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit c418dd720b9be9047d779f32407c474e90cb0002)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 7.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 6.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn 5.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn queens-eol release.