Timeout in the scenario jobs' execution

[Slawek Kaplonski]

I noticed pretty many times when neutron-tempest-plugin scenario jobs were timing out due to very slow tests execution. See examples:

    https://82f03bf860caa31b7ef2-7540de26d324888abf6b8e200e8c6ffb.ssl.cf5.rackcdn.com/816800/5/check/neutron-tempest-plugin-scenario-openvswitch/fa11d7a/job-output.txt

    https://f4a78187a8f66e46939f-e2f3a8f1da38bd85104d6de65559a608.ssl.cf1.rackcdn.com/819032/3/check/neutron-tempest-plugin-scenario-linuxbridge/626868b/job-output.txt

    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_38c/816802/7/check/neutron-tempest-plugin-scenario-linuxbridge/38cc377/job-output.txt

    https://eddc2f34a262a7fd8f98-de6b79a0bbc85dd849a2bc7008d89fe0.ssl.cf1.rackcdn.com/820125/1/gate/neutron-tempest-plugin-scenario-linuxbridge/a49f9a4/job-output.txt

    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1d3/815855/17/gate/neutron-tempest-plugin-scenario-openvswitch/1d38034/job-output.txt

    https://18df87e68d1f57858165-9339bc0d426f2a9334b357b553bd2c47.ssl.cf2.rackcdn.com/815855/17/gate/neutron-tempest-plugin-scenario-ovn/1c40e32/job-output.txt

    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_430/804846/18/check/neutron-tempest-plugin-scenario-ovn/43061b5/job-output.txt

[Balazs Gibizer (balazs-gibizer)]

Here are a couple of recent slow tests leading to job timeout https://paste.opendev.org/show/811635/

[Slawek Kaplonski (slaweq)]

Another examples:

https://4b558001c4cbf62a997f-78e633edf2b137bdab04e16fad5df952.ssl.cf1.rackcdn.com/821433/2/check/neutron-tempest-plugin-scenario-linuxbridge/315a4ed/job-output.txt

https://31dc06b9181c81ffc168-58cfb50a2359d1cd86665eaaa1313c86.ssl.cf5.rackcdn.com/821433/2/check/neutron-tempest-plugin-scenario-openvswitch/9b1b98a/job-output.txt

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/821067
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/165e40922470ae26f9bd99fb28e51abf832ee798
Submitter: "Zuul (22348)"
Branch: master

commit 165e40922470ae26f9bd99fb28e51abf832ee798
Author: yatinkarel <email address hidden>
Date: Wed Dec 8 19:19:34 2021 +0530

    Switch scenario jobs to nested-virt nodes

    To avoid intermittent failures and job timeouts
    in scenario jobs, move these jobs to nested virt nodes.

    Only moving scenario jobs as those are mostly affected
    by intermittent issues(SSH Tempest failures) and job
    timeouts. Also nested virt nodes are provided only by a
    few nodepool providers.

    Also switching non-scenario jobs to use cirros uec
    image to avoid kernel panic issue.

    Initial tests were done with [2] and issues/improvements
    are being tracked at [3].

    [1] https://bugs.launchpad.net/nova/+bug/1939108
    [2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/819590
    [3] https://etherpad.opendev.org/p/neutron-ci-improvements

    Related-Bug: #1952066
    Related-Bug: #1953479
    Change-Id: I2f78f9de1ad1dc8c34688951c6bb2d5648d5dc3f