neutron-tempest-plugin tests fail for stable/queens

[Bernard Cafarelli]

Recent stable/queens backport fail on neutron-tempest-plugin scenario tests, sample here:
https://c896d480cfbd9dee637c-6e2dfe610262db0cf157ed36bc183b08.ssl.cf2.rackcdn.com/688719/5/check/neutron-tempest-plugin-scenario-openvswitch-queens/f080f61/testr_results.html

Traceback (most recent call last):
  File "tempest/common/utils/__init__.py", line 108, in wrapper
    return func(*func_args, **func_kwargs)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/test_internal_dns.py", line 72, in test_dns_domain_and_name
    timeout=CONF.validation.ping_timeout * 10)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/base.py", line 309, in check_remote_connectivity
    timeout=timeout))
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/base.py", line 299, in _check_remote_connectivity
    ping_remote, timeout or CONF.validation.ping_timeout, 1)
  File "tempest/lib/common/utils/test_utils.py", line 107, in call_until_true
    if func(*args, **kwargs):
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/base.py", line 283, in ping_remote
    fragmentation=fragmentation)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/base.py", line 278, in ping_host
    return source.exec_command(cmd)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/tenacity/__init__.py", line 311, in wrapped_f
    return self.call(f, *args, **kw)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/tenacity/__init__.py", line 391, in call
    do = self.iter(retry_state=retry_state)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/tenacity/__init__.py", line 338, in iter
    return fut.result()
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/tenacity/__init__.py", line 394, in call
    result = fn(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/neutron_tempest_plugin/common/ssh.py", line 205, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
  File "tempest/lib/common/ssh.py", line 153, in exec_command
    with transport.open_session() as channel:
AttributeError: 'NoneType' object has no attribute 'open_session'

Queens jobs were pinned to 0.7.0 version of the plugin in a4962ec62808fc469eaad73b1408447d8e3bc7ec it looks like we now need to also pin tempest itself to a "queens version"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702868

[Bernard Cafarelli (bcafarel)]

Testing forcing queens-em tag for tempest in https://review.opendev.org/702868

[Ghanshyam Mann (ghanshyammann)]

which change in queens s incompatible? that will help to understand the issue completely.

Also queens is EM now and I am in progress to officially end the support of queens in Tempest[1]. After doing the release for Tempest, and based on change broke the Tempest queens we can pin the Tempest on devstack like we did for pike and ocata. There we can handle the upper-contraint things also as i mentioned in other bug (https://bugs.launchpad.net/neutron/+bug/1860033) otherwise pining alone would not solve the bug.

[1] https://review.opendev.org/#/c/703255/

[Ghanshyam Mann (ghanshyammann)]

Tempest patch to End the support of EM queens -https://review.opendev.org/#/c/703255/

It will be followed by new Tempest tag release.

[Bernard Cafarelli (bcafarel)]

Thanks for all the help :) Yes, pinning is not enough as it then shows the neutron-lib python2 EOL issue (sorry I only commented on the review about this)

Relevant change in tempest is the one for bug 1853264 [1], which was fixed in neutron-tempest-plugin in bug 1858260 [2] - but this was after latest tag for queens (used as pinned plugin version for this branch tests)

[1] https://review.opendev.org/695167
[2] https://review.opendev.org/701018/

[Ghanshyam Mann (ghanshyammann)]

ok. Using different version of plugin and Tempest can cause even more issue. We should not only pin plugins without pinning Tempest on devstack stable/xyz branch. Also pinning only neutron-tempest-plugin is risky from upper-contraint point of view also. Tempest venv (which install plugins also) use master u-c. so using Temepst master + master u-c + old plugins can break anytime.

We can solve this by two ways:
1. use master neutron-tempest-plugin the same as done for Tempest.
2. pin Tempest and plugins alogn with upper-contraints on devstack stable/queens.

I am not aware of exact reason about pining neutron-tempest-plugin. Policy about stable branch testing even they are in EM state is to use master Tempest & plugins. Once master Tempest or plugins start failing on EM stable for valid reason then, pin those.

We are hitting py2 drop issue on stable/queens and it is EM so its time to pin Tempest & plugins with u-c on devstack which is option2.

Once Tempest 23.0.0 release[1] is done then I will push patch on devstack stable/queens.

[1] https://review.opendev.org/#/c/703521/

[Bernard Cafarelli (bcafarel)]

Ack, noted for next EM release.

So for queens, once devstack patch is we will have tempest itself pinned, we should pin plugin in job to newer revision and clean tempest branch forced by:
https://opendev.org/openstack/neutron-tempest-plugin/commit/a4962ec62808fc469eaad73b1408447d8e3bc7ec

And everything should be back in passing order?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/703679

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/703903

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bernard Cafarelli (<email address hidden>) on branch: master
Review: https://review.opendev.org/702868
Reason: Full fix in progress at https://review.opendev.org/#/c/703903/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/704049

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703679
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=4014fb8c5237701164b1422292aa611ad8fc741e
Submitter: Zuul
Branch: stable/queens

commit 4014fb8c5237701164b1422292aa611ad8fc741e
Author: Ghanshyam <email address hidden>
Date: Tue Jan 21 13:22:21 2020 -0600

    Use Tempest v21.0.0 for Queens testing instead of master

    Tempest and master and neutron-tempest-plugin queens tag is not
    compatible for stable/queens testing.
    More details on https://bugs.launchpad.net/tempest/+bug/1859988

    Also py2 drop will start required the python 3.6 version for Tempest
    master tests.

    Because stable/queens is EM, let's pin the Tempest for its tesing
    instead of putting more effort to make Tempest master run-able.

    Tempest 21.0.0 is supported Tag for Queens so let's use that for Queens
    testing instead of Tempest master.

    Closes-Bug: 1859988

    [1] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html

    Change-Id: Id4861830c46867b313a5a705fc4722ac13471777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704049
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=1b63b3d7aa5b3b446c62cf9137106ba4d9230e7a
Submitter: Zuul
Branch: master

commit 1b63b3d7aa5b3b446c62cf9137106ba4d9230e7a
Author: Ghanshyam <email address hidden>
Date: Thu Jan 23 12:00:04 2020 -0600

    Fix neutron-tempest-plugin tag for queens jobs

    neutron-tempest-plugin tag 0.3.0 is for stable/queen which is
    compatible with neutron-lib==1.13.0[1].

    We are pinning Tempest and upper constraint used in tempest venv
    for stable/queens which pickup neutron-lib as 1.13.0 from stable/queens
    upper constraint.

    Currently used tag of neutron-tempest-plugin 0.7.0 for queens job
    is not compatible with stable/queens neutron-lib version. we can see
    the failure[2]

    Also skip the dns test for queens job which fixed went in 0.4.0 tag
    (https://bugs.launchpad.net/neutron/+bug/1826419) and 0.4.0 tag is not
    compatible with stable/queens constraint.

    Related-Bug: 1859988

    [1]https://github.com/openstack/requirements/blob/f7b45b47bfc5f9b635eb5fa6613482c8c253b091/upper-constraints.txt#L58

    [2]
    - https://6631124a20df425843c7-d8780b91d5409425c6866f467a1249cf.ssl.cf5.rackcdn.com/703680/3/check/neutron-tempest-plugin-api-queens/bba5f0e/job-output.txt

    2020-01-23 17:36:50.747528 | controller | File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/bgpvpn/scenario/test_bgpvpn_basic.py", line 20, in <module>
    2020-01-23 17:36:50.747537 | controller | from neutron_lib.utils import test
    2020-01-23 17:36:50.747545 | controller | ImportError: cannot import name test

    Change-Id: Icc4a91c333ac746183cb423f5e6c59ee0fbdfe2f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703903
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=0327bc227b7aa15f15328b32309fecd506a27830
Submitter: Zuul
Branch: master

commit 0327bc227b7aa15f15328b32309fecd506a27830
Author: Ghanshyam <email address hidden>
Date: Wed Jan 22 18:23:15 2020 -0600

    Remove TEMPEST_BRANCH var from neutron queens job.

    In devstack install_Tempest, ref is taken by TEMPEST_BRANCH var[1]. TEMPEST_BRANCH var on zuul job
    definitoin is priority than the default value in devstack[2]. When we set the queens compatible Tempest
    tag[1] (with review style not by tag name) on devstack stable/queens, that is not taken by devstack
    because neutron queens job define the smae with different value[4].

    Also Tag name (for example queens-em) never work because of devstack logic. Tempest tag name only work
    if RECLONE is true otherwise it should be gerrit ref style. So always Tempest master is installed
    on queens jobs.

    Let's remove the TEMPEST_BRANCH from neutron jobs and rely on devstack deafult value.

    CLoses-Bug: 1859988
    [1] https://github.com/openstack/devstack/blob/stable/queens/lib/tempest#L658
    [2] https://github.com/openstack/devstack/blob/786c485cf6d664163b4ee3d390c62693a9e0d036/stackrc#L313
    [3] https://review.opendev.org/#/c/703679/
    [4] https://zuul.opendev.org/t/openstack/build/0cc32b278afe4e51b787d87d480f7092/log/job-output.txt#12370

    Change-Id: I07beb6e7bdcbaf7ffe72d47f9edb90df40641e68

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 0.8.0 release.