Error: serevr <id> failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None on two tempest test and that failed featureset020-master job

[Bhagyashri Shewale]

"periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master" job is consistently failing here https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master because of some tempest test failures.

Below are the Failed test cases with some error traceback:

{1} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration [423.676137s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 227, in test_server_connectivity_cold_migration
        'VERIFY_RESIZE')
      File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
        raise lib_exc.TimeoutException(message)
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: (TestNetworkAdvancedServerOps:test_server_connectivity_cold_migration) Server 5f65ba3e-5dd2-4c96-81fd-03e843691dd4 failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None.

{1} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize [426.035365s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_advanced_server_ops.py", line 194, in test_server_connectivity_resize
        'VERIFY_RESIZE')
      File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
        raise lib_exc.TimeoutException(message)
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: (TestNetworkAdvancedServerOps:test_server_connectivity_resize) Server 59e35f4e-ed51-4b3d-bb20-fc96f31338fe failed to reach VERIFY_RESIZE status and task state "None" within the required time (300 s). Current status: ACTIVE. Current task state: None.

Note: As the above test are consistently failing and that caused job failure so will move these test cases in the tempest skip list.

Reference link:

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/a60cfb9/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/a60cfb9/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/a60cfb9/logs/undercloud/var/log/extra/errors.txt.txt.gz

[wes hayutin (weshayutin)]

@Bhagyashri

w/ regards to fs020 ( a full tempest run ) The goal w/ fs020 is to open a lp for each root cause of a tempest failure ( root cause being as best known at the time )

The lp should be specific and actionable. Meaning I can add at least one test to the tempest skip list based on the bug.

skip list (new): https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip_master.yml

skip list (current):
https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_master.yml

For example: https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=tempest&field.status%3Alist=TRIAGED&field.status%3Alist=FIXRELEASED&assignee_option=any&field.assignee=&field.bug_reporter=weshayutin&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search&orderby=-datecreated&start=0

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/727192
Reason: no longer needed.. os-ansible-tempest and ansible-role-tempest-skip should handle it.