networking-ovn

random failures of neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle

Bug #1845479 reported by Maciej Jozefczyk on 2019-09-26
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
​ networking-ovn
Fix Released
Critical
Maciej Jozefczyk
Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

Tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle randomly fails on master and stable branches.

2019-09-26 08:22:58.803569 | primary | {2} neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle [145.434088s] ... FAILED
2019-09-26 08:22:58.804121 | primary |
2019-09-26 08:22:58.804180 | primary | Captured traceback:
2019-09-26 08:22:58.804250 | primary | ~~~~~~~~~~~~~~~~~~~
2019-09-26 08:22:58.804331 | primary | b'Traceback (most recent call last):'
2019-09-26 08:22:58.804475 | primary | b' File "/opt/stack/new/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 78, in wait_until_true'
2019-09-26 08:22:58.804615 | primary | b' eventlet.sleep(sleep)'
2019-09-26 08:22:58.804760 | primary | b' File "/usr/local/lib/python3.5/dist-packages/eventlet/greenthread.py", line 34, in sleep'
2019-09-26 08:22:58.804834 | primary | b' hub.switch()'
2019-09-26 08:22:58.804967 | primary | b' File "/usr/local/lib/python3.5/dist-packages/eventlet/hubs/hub.py", line 294, in switch'
2019-09-26 08:22:58.805029 | primary | b' return self.greenlet.switch()'
2019-09-26 08:22:58.805085 | primary | b'eventlet.timeout.Timeout: 60 seconds'
2019-09-26 08:22:58.805108 | primary | b''
2019-09-26 08:22:58.805194 | primary | b'During handling of the above exception, another exception occurred:'
2019-09-26 08:22:58.805217 | primary | b''
2019-09-26 08:22:58.805271 | primary | b'Traceback (most recent call last):'
2019-09-26 08:22:58.805420 | primary | b' File "/opt/stack/new/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 236, in test_trunk_subport_lifecycle'
2019-09-26 08:22:58.805504 | primary | b' self._wait_for_trunk(vm.trunk)'
2019-09-26 08:22:58.805662 | primary | b' File "/opt/stack/new/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 148, in _wait_for_trunk'
2019-09-26 08:22:58.805762 | primary | b' "status {!r}.".format(trunk[\'id\'], status)))'
2019-09-26 08:22:58.805899 | primary | b' File "/opt/stack/new/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 82, in wait_until_true'
2019-09-26 08:22:58.805940 | primary | b' raise exception'
2019-09-26 08:22:58.806074 | primary | b"RuntimeError: Timed out waiting for trunk 'a2867f25-6ce0-4596-8517-793f0e804c5c' to transition to get status 'ACTIVE'."
2019-09-26 08:22:58.806134 | primary | b''
2019-09-26 08:22:58.806161 | primary |
-------------------------------------------------------------------------------

While checking gates logs I found this error in neutron logs:

-----------------------------------------------------------------------------
Sep 26 08:21:51.753519 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: DEBUG neutron.db.provisioning_blocks [None req-029df57c-a96e-4274-9a3f-54bb050d655d None None] Transition to ACTIVE for port object 5bf4e97f-71cb-43e4-9957-ce38a52c97c7 will not be triggered until provisioned by entity L2. {{(pid=31758) add_provisioning_component /opt/stack/new/neutron/neutron/db/provisioning_blocks.py:73}}
Sep 26 08:21:52.172865 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: DEBUG neutron_lib.callbacks.manager [None req-267c86f8-eb08-4076-833f-53a16eab185c tempest-TrunkTest-1452780584 tempest-TrunkTest-1452780584] Notify callbacks [] for subports, precommit_create {{(pid=31756) _notify_loop /usr/local/lib/python3.5/dist-packages/neutron_lib/callbacks/manager.py:193}}
Sep 26 08:21:52.176038 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: DEBUG neutron_lib.callbacks.manager [None req-267c86f8-eb08-4076-833f-53a16eab185c tempest-TrunkTest-1452780584 tempest-TrunkTest-1452780584] Notify callbacks ['networking_ovn.ml2.trunk_driver.OVNTrunkHandler.subport_event--9223372036854446820'] for subports, after_create {{(pid=31756) _notify_loop /usr/local/lib/python3.5/dist-packages/neutron_lib/callbacks/manager.py:193}}
Sep 26 08:21:52.338192 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager [None req-267c86f8-eb08-4076-833f-53a16eab185c tempest-TrunkTest-1452780584 tempest-TrunkTest-1452780584] Error during notification for networking_ovn.ml2.trunk_driver.OVNTrunkHandler.subport_event--9223372036854446820 subports, after_create: sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 26 08:21:52.338524 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Sep 26 08:21:52.338768 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Sep 26 08:21:52.339033 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Sep 26 08:21:52.339249 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/networking-ovn/networking_ovn/ml2/trunk_driver.py", line 145, in subport_event
Sep 26 08:21:52.339552 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager payload.subports)
Sep 26 08:21:52.339771 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/networking-ovn/networking_ovn/ml2/trunk_driver.py", line 128, in subports_added
Sep 26 08:21:52.340031 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager self._set_sub_ports(trunk.port_id, subports)
Sep 26 08:21:52.340241 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/networking-ovn/networking_ovn/ml2/trunk_driver.py", line 50, in _set_sub_ports
Sep 26 08:21:52.340449 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager self._set_binding_profile(port, parent_port, ovn_txn)
Sep 26 08:21:52.340712 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/networking-ovn/networking_ovn/ml2/trunk_driver.py", line 75, in _set_binding_profile
Sep 26 08:21:52.340934 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager host=binding.host)
Sep 26 08:21:52.341142 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/base.py", line 675, in update_object
Sep 26 08:21:52.341394 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager **cls.modify_fields_to_db(kwargs))
Sep 26 08:21:52.341652 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 82, in update_object
Sep 26 08:21:52.341915 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager db_obj.save(session=context.session)
Sep 26 08:21:52.342149 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/oslo_db/sqlalchemy/models.py", line 50, in save
Sep 26 08:21:52.342380 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager session.flush()
Sep 26 08:21:52.342657 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2254, in flush
Sep 26 08:21:52.342870 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager self._flush(objects)
Sep 26 08:21:52.343092 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2380, in _flush
Sep 26 08:21:52.343345 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager transaction.rollback(_capture_exception=True)
Sep 26 08:21:52.343542 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
Sep 26 08:21:52.343761 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager compat.reraise(exc_type, exc_value, exc_tb)
Sep 26 08:21:52.344039 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 249, in reraise
Sep 26 08:21:52.344263 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager raise value
Sep 26 08:21:52.344488 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2344, in _flush
Sep 26 08:21:52.344800 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager flush_context.execute()
Sep 26 08:21:52.345069 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
Sep 26 08:21:52.345384 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager rec.execute(self)
Sep 26 08:21:52.345724 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
Sep 26 08:21:52.347471 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager uow
Sep 26 08:21:52.348294 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
Sep 26 08:21:52.348651 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager mapper, table, update)
Sep 26 08:21:52.348943 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 791, in _emit_update_statements
Sep 26 08:21:52.349248 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager (table.description, len(records), rows))
Sep 26 08:21:52.349537 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 26 08:21:52.349836 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: ERROR neutron_lib.callbacks.manager
Sep 26 08:21:52.350105 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: INFO neutron.wsgi [None req-267c86f8-eb08-4076-833f-53a16eab185c tempest-TrunkTest-1452780584 tempest-TrunkTest-1452780584] 158.69.66.180 "PUT /v2.0/trunks/a2867f25-6ce0-4596-8517-793f0e804c5c/add_subports HTTP/1.1" status: 200 len: 1004 time: 0.9139748
Sep 26 08:21:52.350384 ubuntu-xenial-ovh-bhs1-0011869985 neutron-server[31580]: DEBUG neutron.wsgi [-] (31756) accepted ('158.69.66.180', 41394) {{(pid=31756) server /usr/local/lib/python3.5/dist-packages/eventlet/wsgi.py:883}}

See original description
Maciej Jozefczyk (maciejjozefczyk) on 2019-09-26
description: updated
Changed in networking-ovn:
assignee: nobody → Maciej Jozefczyk (maciej.jozefczyk)
status: New → In Progress
Maciej Jozefczyk (maciejjozefczyk) wrote :

Example run:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c46/683928/1/check/networking-ovn-tempest-dsvm-ovs-release-python3/c46ac38/job-output.txt

Example error:
https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c46/683928/1/check/networking-ovn-tempest-dsvm-ovs-release-python3/c46ac38/logs/screen-q-svc.txt.gz

Maciej Jozefczyk (maciejjozefczyk) wrote :

Statistics taken from status.openstack.org:
networking-ovn-tempest-dsvm-ovs-release

Test Name Passes Failures Skips % Failures Mean Runtime
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle 25 4 0 13.79 79.10

networking-ovn-tempest-dsvm-ovs-release-python3
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle 5 1 0 16.67 96.06

I think we should revert it

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/685313

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/685315

OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-ovn (master)

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.opendev.org/685313
Reason: This was supposed to go to stable/train

Jakub Libosvar (libosvar) on 2019-10-01
Changed in networking-ovn:
importance: Undecided → Critical
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (master)

Reviewed: https://review.opendev.org/685714
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=733e1d672648943f5d0307e1fdb3eb4abb381b53
Submitter: Zuul
Branch: master

commit 733e1d672648943f5d0307e1fdb3eb4abb381b53
Author: Maciej Józefczyk <email address hidden>
Date: Mon Sep 30 15:41:54 2019 +0200

    Temporary allow to fail Trunk rally tests

    We have stability issues with Trunk driver optimalizations [0].
    In meantime neutron rally jobs sync has been merged [1] that
    enabled Trunk scenarios.
    Lets allow those rally tests failures, rollback [0] and in meantime fix it.

    [0] https://review.opendev.org/#/c/676182/
    [1] https://review.opendev.org/#/c/664605/

    Related-Bug: #1845479

    Change-Id: Icf0f0f42f02180c787159fbd740b54316d5db997

OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/685206
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=2e0832f7b8bfc31780b657aa0abda4e8b244fbbd
Submitter: Zuul
Branch: master

commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:46:40 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 41f6d622abc65ca4c10b08f8c022915df2809255.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I63acf0c809aa158aeba1c79c7c9083966ffd58a3

OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/train)

Reviewed: https://review.opendev.org/685315
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=60ca3c7809cba359871ed7220715a637448d9362
Submitter: Zuul
Branch: stable/train

commit 60ca3c7809cba359871ed7220715a637448d9362
Author: Jakub Libosvar <email address hidden>
Date: Fri Sep 27 12:23:09 2019 +0200

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 41f6d622abc65ca4c10b08f8c022915df2809255.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I244b937e916474703abf0a0c802e13bcd5a54918

tags: added: in-stable-train
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/stein)

Reviewed: https://review.opendev.org/685209
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=4f2301a060ea8ce941b6a9c48d731537ce32494a
Submitter: Zuul
Branch: stable/stein

commit 4f2301a060ea8ce941b6a9c48d731537ce32494a
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:50:19 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 1d58b0e680ee3da65a58620639b220bd972318fe.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I55d224faead40da99b9c9f8ded5117a9b0bc288c

tags: added: in-stable-stein
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/rocky)

Reviewed: https://review.opendev.org/685207
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=3205f887fcc9a509c1a281f2d6d3788ff5147ba4
Submitter: Zuul
Branch: stable/rocky

commit 3205f887fcc9a509c1a281f2d6d3788ff5147ba4
Author: Maciej Józefczyk <email address hidden>
Date: Fri Sep 27 06:47:10 2019 +0000

    Revert "Set binding profile directly from OVNTrunkDriver"

    During recent days we many observed failures of
    neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle test.
    It started to fail on a different asserts.
    Lets revert this patch, as it is a regression, and work further on this issue.

    This reverts commit 1e2e424f5d90be32edaf701d1871701bf4b5d522.

    Related-Bug: #1834637
    Related-Bug: #1845479

    Change-Id: I862751fabcd590e91d0ded00dd1cfeb9e6753f2f

tags: added: in-stable-rocky
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (master)

Reviewed: https://review.opendev.org/686571
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=7bdf2eb824083b6785d78dd67b3effe071e3f7a4
Submitter: Zuul
Branch: master

commit 7bdf2eb824083b6785d78dd67b3effe071e3f7a4
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/687919

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/687930

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687971

OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-ovn (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/687972

Bernard Cafarelli (bcafarel) on 2019-10-11
tags: added: networking-ovn-proactive-backport-potential
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (master)

Reviewed: https://review.opendev.org/687505
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=5d59ec7a18f89e05d067c2538b1854756af271b5
Submitter: Zuul
Branch: master

commit 5d59ec7a18f89e05d067c2538b1854756af271b5
Author: Flavio Fernandes <email address hidden>
Date: Wed Oct 9 09:17:30 2019 +0000

    Revert "Temporary allow to fail Trunk rally tests"

    This reverts commit 733e1d672648943f5d0307e1fdb3eb4abb381b53.

    That is no longer needed with the merge of
    https://review.opendev.org/#/c/686571/

    Change-Id: I85f3d51dd7066b8be6adfc183f120d3d3b4f1e77
    Closes-Bug: #1845479

Changed in networking-ovn:
status: In Progress → Fix Released
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/train)

Reviewed: https://review.opendev.org/687919
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1e20b9bf866e3001ad1cd71186caada647c54929
Submitter: Zuul
Branch: stable/train

commit 1e20b9bf866e3001ad1cd71186caada647c54929
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    ---
    Also, merge squashed:

    Enable ovs-vswitchd and ovsdb-server for rally

    The ovs-vswitchd and ovsdb-server services were missing in the rally
    definition. Usually the jobs inherts from our base class which enables
    them but rally has a different job parent.

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    Co-authored-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 7bdf2eb824083b6785d78dd67b3effe071e3f7a4)
    (cherry picked from commit 202e804c64c9c6d7160496987a4b13bd9f9ee6b6)
    (cherry picked from commit 7e725331dbaf5845d6a45fa1cf8035d6a61ac744)

OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/stein)

Reviewed: https://review.opendev.org/687930
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=c6dee38266b662a2bb90c95af891045c731daf41
Submitter: Zuul
Branch: stable/stein

commit c6dee38266b662a2bb90c95af891045c731daf41
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit 1e20b9bf866e3001ad1cd71186caada647c54929)

OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/queens)

Reviewed: https://review.opendev.org/687972
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=01d86c3c97b196279769522532ccdef2c3bf52ec
Submitter: Zuul
Branch: stable/queens

commit 01d86c3c97b196279769522532ccdef2c3bf52ec
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Note: While this is a back-merge from Rocky, special care was
    needed to account for the fact that in Queens release there
    is only one port binding for a db_port. Thus no iteration over
    db_port.bindings is used.

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit 62eb828186f9248e99fa12cc9c0c8397e1520514)

tags: added: in-stable-queens
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-ovn (stable/rocky)

Reviewed: https://review.opendev.org/687971
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=62eb828186f9248e99fa12cc9c0c8397e1520514
Submitter: Zuul
Branch: stable/rocky

commit 62eb828186f9248e99fa12cc9c0c8397e1520514
Author: Flavio Fernandes <email address hidden>
Date: Thu Sep 5 09:29:45 2019 +0000

    Set binding profile directly from OVNTrunkDriver (redo)

    Setting binding profile for Trunk subports takes
    time - for 125 subports rally CreateAndListTrunks
    scenario [0] takes about 150 seconds. We need to
    bump up the perfomance because large number of
    subports is widly used in Kuryr deployments.

    To achieve that I changed setting the binding
    profile to be saved directly to the neutron DB.
    Instead calling port_update I update only related
    fields in OVN NorthBound DB rows. That gave performance
    improvement in trunk port creation:

    from 101 sec to 35.6 for 95%ile
    from 99 sec to 34.2 for 50%ile

    The same thing has been done for Trunk deletion.

    This reverts commit 2e0832f7b8bfc31780b657aa0abda4e8b244fbbd

    [0] https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/neutron/trunk.py#L37

    Change-Id: I6b659cbede25f271fa3b6a1c9e72019694ab6608
    Closes-Bug: #1834637
    Related-Bug: #1845479
    Co-authored-by: Maciej Józefczyk <email address hidden>
    (cherry picked from commit c6dee38266b662a2bb90c95af891045c731daf41)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.
Subscribing...

Other bug subscribers