Some tempest tests are unstable

Bug #1736685 reported by Dougal Matthews
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Won't Fix
Low
Dougal Matthews

Bug Description

The tempest CI jobs are not currently voting because they are unstable. We should fix this!

We need to collect a list of unstable tests, and disable them. Then enable voting on the jobs and then fix or replace the tests that were unstable.

I am using the rationale, that if a test is seen failing in two unrelated patches then it is unstable or rechecking a patch causes it to sporadically pass and fail.

This is something of a meta-bug. We may want to open individual bugs for each of the test failures.

Tags: tripleo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/526002

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in https://review.openstack.org/#/c/520218/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync

description: updated
Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in https://review.openstack.org/#/c/506185/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog

description: updated
description: updated
Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in

mistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_in_list_filter

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog [SECOND UNRELATED FAILURE. DISABLING: https://review.openstack.org/526002 ]

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/526034

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in https://review.openstack.org/#/c/526002/

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync [SECOND UNRELATED FAILURE. DISABLING: https://review.openstack.org/526034/ ]

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync

Dougal Matthews (d0ugal)
description: updated
Revision history for this message
Brad P. Crochet (brad-9) wrote :

Definitely create a bug for each failure.

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Bugs for the two current tests with patches up to disable them:

https://bugs.launchpad.net/mistral/+bug/1736931
https://bugs.launchpad.net/mistral/+bug/1736933

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral (master)

Reviewed: https://review.openstack.org/526002
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=cd37173c8c1134155e9218b67e4a68c56e79d77c
Submitter: Zuul
Branch: master

commit cd37173c8c1134155e9218b67e4a68c56e79d77c
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 09:24:16 2017 +0000

    Disable unstable tempest multi_vim_authentication test

    This disables test_multi_vim_support_target_headers_and_service_catalog.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Ie6a9e67837fb8c6386f915209ecf11974ab052c6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/526034
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=d53c1a983454777f8bee16806a1331bb0bd5c1cb
Submitter: Zuul
Branch: master

commit d53c1a983454777f8bee16806a1331bb0bd5c1cb
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 11:23:09 2017 +0000

    Disable unstable tempest test_create_action_execution_sync test

    This disables test_create_action_execution_sync.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Icfa134d391476cad6ef925c84e93754d8a625ef2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/522880
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=b084b6edd735f1f7a01885c19a994fb1f13455ae
Submitter: Zuul
Branch: master

commit b084b6edd735f1f7a01885c19a994fb1f13455ae
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 24 17:10:12 2017 +0000

    Make more CI jobs voting

    A recent regression showed up in the tempest tests, but as they were not
    voting it was ignored and not spotted until later. This caused lots of
    time to be wasted debugging.

    We should make more of the jobs voting and fix any instability (or
    remove particular tests that are unstable).

    Related-Bug: #1736685
    Change-Id: Ifd6f1d659db06bca3e6417b9235258fd02f1da88

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/526082
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866
Submitter: Zuul
Branch: master

commit b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 14:17:42 2017 +0000

    Re-work the direct action call tempest test

    The current version calls the Mistral API, this seems to be unstable.
    Using another std action is enough to validate that this works
    correctly.

    Related-Bug: #1736685
    Change-Id: I2ea70b228f3240ed89c3a3d4bd991459a6736f8c

Dougal Matthews (d0ugal)
Changed in mistral:
assignee: nobody → Dougal Matthews (d0ugal)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral-tempest-plugin (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/527641

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/527642

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/527643

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral-tempest-plugin (master)

Reviewed: https://review.openstack.org/527641
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=955327c231ba9e837b10abb3fae5b991e66cda2d
Submitter: Zuul
Branch: master

commit 955327c231ba9e837b10abb3fae5b991e66cda2d
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 09:24:16 2017 +0000

    Disable unstable tempest multi_vim_authentication test

    This disables test_multi_vim_support_target_headers_and_service_catalog.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Ie6a9e67837fb8c6386f915209ecf11974ab052c6
    (cherry picked from commit cd37173c8c1134155e9218b67e4a68c56e79d77c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/527642
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=0dc7e574b32d8898cd3196484dd0035100b4bc77
Submitter: Zuul
Branch: master

commit 0dc7e574b32d8898cd3196484dd0035100b4bc77
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 11:23:09 2017 +0000

    Disable unstable tempest test_create_action_execution_sync test

    This disables test_create_action_execution_sync.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Icfa134d391476cad6ef925c84e93754d8a625ef2
    (cherry picked from commit d53c1a983454777f8bee16806a1331bb0bd5c1cb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/527643
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=2b4471bf84eb858ab90569c60de3c2ebe3f56cc5
Submitter: Zuul
Branch: master

commit 2b4471bf84eb858ab90569c60de3c2ebe3f56cc5
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 14:17:42 2017 +0000

    Re-work the direct action call tempest test

    The current version calls the Mistral API, this seems to be unstable.
    Using another std action is enough to validate that this works
    correctly.

    Related-Bug: #1736685
    Change-Id: I2ea70b228f3240ed89c3a3d4bd991459a6736f8c
    (cherry picked from commit b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866)

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in https://review.openstack.org/#/c/525926/ and https://review.openstack.org/#/c/528291/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_ssh_actions.SSHActionsTestsV2.test_run_ssh_proxied_action

Disabling patch incoming...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/528999

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Seen failing in https://review.openstack.org/#/c/525926/ and https://review.openstack.org/#/c/528291/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_ssh_actions.SSHActionsTestsV2.test_run_ssh_proxied_action

Disabling patch incoming...

Revision history for this message
Dougal Matthews (d0ugal) wrote :

I opened a bug for this test failure too. https://bugs.launchpad.net/mistral/+bug/1738974

Revision history for this message
Dougal Matthews (d0ugal) wrote :

See failing in https://review.openstack.org/#/c/528999/

mistral_tempest_tests.tests.api.v2.test_event_triggers.EventTriggerTestsV2.test_get_nonexistent_event_trigger

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to mistral-tempest-plugin (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/530290

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to mistral-tempest-plugin (master)

Reviewed: https://review.openstack.org/530290
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=63a0fe20f98e0cb8316beb81ca77249ffdda29c5
Submitter: Zuul
Branch: master

commit 63a0fe20f98e0cb8316beb81ca77249ffdda29c5
Author: Dougal Matthews <email address hidden>
Date: Tue Dec 19 08:55:27 2017 +0000

    Disable unstable tempest test_run_ssh_proxied_action test

    This disables test_run_ssh_proxied_action. Currently it causes the tests to
    fail frequently which makes them less useful and blocks patches incorrectly.

    Related-Bug: #1736685
    Change-Id: I9bd4cb8f828eca3e3227d8e52ef916191a36b45f

Changed in mistral:
milestone: queens-3 → rocky-2
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-2 → rocky-3
Revision history for this message
Dougal Matthews (d0ugal) wrote :

I think we can close this. I am no longer seeing stability and this bug doesn't have a clear path to fixing/closing

Changed in mistral:
importance: High → Low
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.