create_and_list_trunk_subports rally scenario failed with timeouts

[Ihar Hrachyshka]

This happened once in Pike.

http://logs.openstack.org/16/529616/1/check/legacy-rally-dsvm-neutron-neutron/72b511a/rally-plot/results.html.gz#/NeutronTrunks.create_and_list_trunk_subports/failures

Traceback (most recent call last):
  File "/opt/stack/new/rally/rally/task/runner.py", line 71, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 41, in run
    trunk = self._create_trunk(trunk_payload)
  File "/opt/stack/new/rally/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/home/zuul/.rally/plugins/plugins/trunk_scenario.py", line 53, in _create_trunk
    return self.clients("neutron").create_trunk({'trunk': trunk_payload})
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 2067, in create_trunk
    return self.post(self.trunks_path, body=body)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 357, in post
    headers=headers, params=params)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 342, in do_request
    return self.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 330, in request
    resp = super(SessionClient, self).request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 192, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 108, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request
    resp = send(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 768, in _send_request
    raise exceptions.ConnectTimeout(msg)
ConnectTimeout: Request to http://104.239.168.166:9696/v2.0/trunks timed out

All four attempts failed the same way. Other scenarios worked just fine.

When you look at q-svc log, you see that all four POST requests for trunks took a very long time, for example, req-b98d273b-8c38-4d54-98f5-1d74008795d8 took 3:30 minutes to get from quotas check to precommit_create notification:

http://logs.openstack.org/16/529616/1/check/legacy-rally-dsvm-neutron-neutron/72b511a/logs/screen-q-svc.txt.gz?#_Jan_05_19_46_52_971085

The only major code between quota check and this notification is validation of the request. Requests are, granted, heavy loaded, with 125 subports per trunk, so it may have something to do with it.

[Ihar Hrachyshka (ihar-hrachyshka)]

As Armando pointed out, even in 'good' runs those scenarios take a lot of time, so we probably just hit the edge of the timeout allowed by neutronclient. We may optimize db operations when validating trunk requests, specifically, we could fetch all networks at once to validate MTU compatibility, offloading bulking to db native layer.

We have a similar code that bulk-fetches networks here: https://github.com/openstack/neutron/blob/master/neutron/services/trunk/rules.py#L191 that we may somewhat reuse.

[Armando Migliaccio (armando-migliaccio)]

While it looks like runs have generally become slower, there's an opportunity to speed things up to avoid fetching MTU's on a per port basis.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/532045

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/532559

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532045
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d23a9ad7313b6d4b275f3b46c89d9f79d3cfb9a4
Submitter: Zuul
Branch: master

commit d23a9ad7313b6d4b275f3b46c89d9f79d3cfb9a4
Author: Armando Migliaccio <email address hidden>
Date: Mon Jan 8 22:46:04 2018 -0800

    Speed up trunk MTU enforcement check

    While things may have become generally slower for reasons still
    to be determined at the time this patch was developed, we noticed
    that enforcing MTU rules between subports and trunk parent was
    done by performing one DB lookup per subport. This patch optmizes
    the performance by piggybacking on the logic that is used to
    retrieve the subport underlying network's segmentation type when
    the segmentation type must be inherited.

    Closes-bug: #1741954

    Change-Id: Ib18427ef7a3509088eaf8fa0cfed3fb659b6baea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532559
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5ab223d1bd533e459ee61572d49aa5e159af22c4
Submitter: Zuul
Branch: stable/pike

commit 5ab223d1bd533e459ee61572d49aa5e159af22c4
Author: Armando Migliaccio <email address hidden>
Date: Mon Jan 8 22:46:04 2018 -0800

    Speed up trunk MTU enforcement check

    While things may have become generally slower for reasons still
    to be determined at the time this patch was developed, we noticed
    that enforcing MTU rules between subports and trunk parent was
    done by performing one DB lookup per subport. This patch optmizes
    the performance by piggybacking on the logic that is used to
    retrieve the subport underlying network's segmentation type when
    the segmentation type must be inherited.

    Closes-bug: #1741954

    Change-Id: Ib18427ef7a3509088eaf8fa0cfed3fb659b6baea
    (cherry picked from commit d23a9ad7313b6d4b275f3b46c89d9f79d3cfb9a4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.3 release.