neutron-rally-task fails in rally_openstack.task.scenarios.neutron.trunk.CreateAndListTrunks

[Bence Romsics]

It seems we have a gate failure in neutron-rally-task. It fails in rally_openstack.task.scenarios.neutron.trunk.CreateAndListTrunks. For example:

https://zuul.opendev.org/t/openstack/build/9c9970da456d4145a174f73c90529dd2/log/job-output.txt#41274
https://zuul.opendev.org/t/openstack/build/8319cc946cc9407a90467f68757c11e8/log/job-output.txt#41269

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%229696%2Fv2.0%2Ftrunks%20timed%20out%5C%22%20AND%20voting:1&from=864000s

[Bence Romsics (bence-romsics)]

We recently merged this trunk-related change so it's a candidate for investigation: https://review.opendev.org/477286

[Bence Romsics (bence-romsics)]

The linked change does mass updates on all subports of a trunk:

            for subport in subports:
                core_plugin.update_port(context, subport.port_id,
                                        {'port': {'device_id': device_id}})

So I would not be suprised to see huge performance effects and timeouts because of that.

[Lajos Katona (lajos-katona)]

As I see the rally task execution time increased significantly:
27. 03. before the merge: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3a5/713290/2/check/neutron-rally-task/3a5c100/results/report.html#/NeutronTrunks.create_and_list_trunks Load duration: 227.115 s Full duration: 408.386 s
this means ~32sec / iteration from html report

01. 04. after the merge:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cea/702247/7/check/neutron-rally-task/cea4f53/results/report.html#/NeutronTrunks.create_and_list_trunks
Load duration: 343.507 s Full duration: 619.916 s
This means ~155sec / iteration

[Lajos Katona (lajos-katona)]

The job is not always failing, and that is due to that we are just hitting the ceiling sometimes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/716562

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716562
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a90654ae5a8f4cc8588f0f595af091f8cd986441
Submitter: Zuul
Branch: master

commit a90654ae5a8f4cc8588f0f595af091f8cd986441
Author: Bence Romsics <email address hidden>
Date: Wed Apr 1 13:30:10 2020 +0200

    Revert "Subcribe trunk & subport events to set subport id"

    This reverts commit 8ebc635a18fd23fd6595551a703961a4d4392948.

    The reverted commit does mass update on all subports of a trunk.
    This is not in line with the original design since it causes huge
    api-side performance effects.

    I think that's the reason why we started seeing gate failures of
    rally_openstack.task.scenarios.neutron.trunk.CreateAndListTrunks
    in neutron-rally-task.

    Change-Id: I6f0fd91c62985207af8dbf29aae463b2b478d5d2
    Closes-Bug: #1870110
    Related-Bug: #1700428