Trunk Tests are failing often in dvr-multinode scenario job

[Slawek Kaplonski]

In about 40% of test runs tests like:
neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle, example runs:
    * http://logs.openstack.org/03/560703/7/check/neutron-tempest-plugin-dvr-multinode-scenario/1f67afd/logs/testr_results.html.gz
    * http://logs.openstack.org/17/553617/19/check/neutron-tempest-plugin-dvr-multinode-scenario/a13a6fd/logs/testr_results.html.gz
    * http://logs.openstack.org/84/533284/5/check/neutron-tempest-plugin-dvr-multinode-scenario/1c09aa6/logs/testr_results.html.gz

neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity, example run:
    * http://logs.openstack.org/90/545490/9/check/neutron-tempest-plugin-dvr-multinode-scenario/c1ed535/logs/testr_results.html.gz

are failing.

[Miguel Lavalle (minsel)]

Using following query:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=%20project:%5C%22openstack%2Fneutron%5C%22%20AND%20build_name:%5C%22neutron-tempest-plugin-dvr-multinode-scenario%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22&from=7d

[Miguel Lavalle (minsel)]

Using:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=%20project:%5C%22openstack%2Fneutron%5C%22%20AND%20message:%5C%22test_trunk_subport_lifecycle%5C%22%20AND%20build_name:%5C%22neutron-tempest-plugin-dvr-multinode-scenario%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22&from=7d

and:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=%20project:%5C%22openstack%2Fneutron%5C%22%20AND%20message:%5C%22test_trunk_subport_lifecycle%5C%22%20AND%20build_name:%5C%22neutron-tempest-plugin-dvr-multinode-scenario%5C%22%20AND%20build_status:%5C%22FAILURE%5C%22&from=7d

I don't find any hist over the past 7 days. I will dig in Gerrit for Zuul failures with this job

[Bernard Cafarelli (bcafarel)]

These tests are currently marked unstable, so will not appear with failed build_status?
https://github.com/openstack/neutron-tempest-plugin/commit/02a5e2b07680d8c4dd69d681ae9a01d92b4be0ac

A recent failure for neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_subport_connectivity: http://logs.openstack.org/73/589573/1/check/neutron-tempest-plugin-dvr-multinode-scenario/b69c365/job-output.txt.gz#_2018-08-08_00_50_01_984287

[Slawek Kaplonski (slaweq)]

Patch proposed for neutron-tempest-plugin: https://review.openstack.org/#/c/602696

[Miguel Lavalle (minsel)]

Fix was merged https://review.openstack.org/#/c/602696

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Do we need to backport his patch?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/611246

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/611247

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/590422
Reason: It's already done in https://review.openstack.org/#/c/602696/3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611247
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f9cf83474971dfc0f47c1ee9b6d5d17583e1bf51
Submitter: Zuul
Branch: stable/ocata

commit f9cf83474971dfc0f47c1ee9b6d5d17583e1bf51
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 12 02:01:31 2018 +0200

    Fix bash cmd used in scenario trunk tests

    In scenario trunk test test_subport_connectivity there
    is bash command used to check name of base interface on
    spawned vm (e.g. ens3 or eth0), configure vlan interface
    with vlan_id used on trunk port (e.g. ens3.10),
    make this vlan device up and run dhclient on this interface
    if it's not running yet.

    This command was broken and that cause failure of this test.

    Change-Id: I4c0207f79cd6df2594f976b9509697209011edf2
    Closes-Bug: #1766701
    (cherry picked from commit 6bf840f0a73579804375dddd92b3a20acc57b877)
    (cherry picked from commit c78b80becfd6ad5865fb1681a05395940591f431)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611246
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c78b80becfd6ad5865fb1681a05395940591f431
Submitter: Zuul
Branch: stable/pike

commit c78b80becfd6ad5865fb1681a05395940591f431
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 12 02:01:31 2018 +0200

    Fix bash cmd used in scenario trunk tests

    In scenario trunk test test_subport_connectivity there
    is bash command used to check name of base interface on
    spawned vm (e.g. ens3 or eth0), configure vlan interface
    with vlan_id used on trunk port (e.g. ens3.10),
    make this vlan device up and run dhclient on this interface
    if it's not running yet.

    This command was broken and that cause failure of this test.

    Change-Id: I4c0207f79cd6df2594f976b9509697209011edf2
    Closes-Bug: #1766701
    (cherry picked from commit 6bf840f0a73579804375dddd92b3a20acc57b877)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 0.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.