Some tempest tests are unstable

[Dougal Matthews]

The tempest CI jobs are not currently voting because they are unstable. We should fix this!

We need to collect a list of unstable tests, and disable them. Then enable voting on the jobs and then fix or replace the tests that were unstable.

I am using the rationale, that if a test is seen failing in two unrelated patches then it is unstable or rechecking a patch causes it to sporadically pass and fail.

This is something of a meta-bug. We may want to open individual bugs for each of the test failures.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/526002

[Dougal Matthews (d0ugal)]

Seen failing in https://review.openstack.org/#/c/520218/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync

[Dougal Matthews (d0ugal)]

Seen failing in https://review.openstack.org/#/c/506185/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog

[Dougal Matthews (d0ugal)]

Seen failing in

mistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_in_list_filter

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_multi_vim_authentication.MultiVimActionsTests.test_multi_vim_support_target_headers_and_service_catalog [SECOND UNRELATED FAILURE. DISABLING: https://review.openstack.org/526002 ]

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/526034

[Dougal Matthews (d0ugal)]

Seen failing in https://review.openstack.org/#/c/526002/

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync [SECOND UNRELATED FAILURE. DISABLING: https://review.openstack.org/526034/ ]

mistral_tempest_tests.tests.api.v2.test_action_executions.ActionExecutionTestsV2.test_create_action_execution_sync

[Brad P. Crochet (brad-9)]

Definitely create a bug for each failure.

[Dougal Matthews (d0ugal)]

Bugs for the two current tests with patches up to disable them:

https://bugs.launchpad.net/mistral/+bug/1736931
https://bugs.launchpad.net/mistral/+bug/1736933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526002
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=cd37173c8c1134155e9218b67e4a68c56e79d77c
Submitter: Zuul
Branch: master

commit cd37173c8c1134155e9218b67e4a68c56e79d77c
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 09:24:16 2017 +0000

    Disable unstable tempest multi_vim_authentication test

    This disables test_multi_vim_support_target_headers_and_service_catalog.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Ie6a9e67837fb8c6386f915209ecf11974ab052c6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526034
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=d53c1a983454777f8bee16806a1331bb0bd5c1cb
Submitter: Zuul
Branch: master

commit d53c1a983454777f8bee16806a1331bb0bd5c1cb
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 11:23:09 2017 +0000

    Disable unstable tempest test_create_action_execution_sync test

    This disables test_create_action_execution_sync.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Icfa134d391476cad6ef925c84e93754d8a625ef2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/522880
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=b084b6edd735f1f7a01885c19a994fb1f13455ae
Submitter: Zuul
Branch: master

commit b084b6edd735f1f7a01885c19a994fb1f13455ae
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 24 17:10:12 2017 +0000

    Make more CI jobs voting

    A recent regression showed up in the tempest tests, but as they were not
    voting it was ignored and not spotted until later. This caused lots of
    time to be wasted debugging.

    We should make more of the jobs voting and fix any instability (or
    remove particular tests that are unstable).

    Related-Bug: #1736685
    Change-Id: Ifd6f1d659db06bca3e6417b9235258fd02f1da88

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526082
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866
Submitter: Zuul
Branch: master

commit b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 14:17:42 2017 +0000

    Re-work the direct action call tempest test

    The current version calls the Mistral API, this seems to be unstable.
    Using another std action is enough to validate that this works
    correctly.

    Related-Bug: #1736685
    Change-Id: I2ea70b228f3240ed89c3a3d4bd991459a6736f8c

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/527641

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/527642

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/527643

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527641
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=955327c231ba9e837b10abb3fae5b991e66cda2d
Submitter: Zuul
Branch: master

commit 955327c231ba9e837b10abb3fae5b991e66cda2d
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 09:24:16 2017 +0000

    Disable unstable tempest multi_vim_authentication test

    This disables test_multi_vim_support_target_headers_and_service_catalog.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Ie6a9e67837fb8c6386f915209ecf11974ab052c6
    (cherry picked from commit cd37173c8c1134155e9218b67e4a68c56e79d77c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527642
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=0dc7e574b32d8898cd3196484dd0035100b4bc77
Submitter: Zuul
Branch: master

commit 0dc7e574b32d8898cd3196484dd0035100b4bc77
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 11:23:09 2017 +0000

    Disable unstable tempest test_create_action_execution_sync test

    This disables test_create_action_execution_sync.
    Doing so allows us to mvoe towards a voting tempest job. We need to come
    back and fix or replace this test later.

    Related-Bug: #1736685
    Change-Id: Icfa134d391476cad6ef925c84e93754d8a625ef2
    (cherry picked from commit d53c1a983454777f8bee16806a1331bb0bd5c1cb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/527643
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=2b4471bf84eb858ab90569c60de3c2ebe3f56cc5
Submitter: Zuul
Branch: master

commit 2b4471bf84eb858ab90569c60de3c2ebe3f56cc5
Author: Dougal Matthews <email address hidden>
Date: Wed Dec 6 14:17:42 2017 +0000

    Re-work the direct action call tempest test

    The current version calls the Mistral API, this seems to be unstable.
    Using another std action is enough to validate that this works
    correctly.

    Related-Bug: #1736685
    Change-Id: I2ea70b228f3240ed89c3a3d4bd991459a6736f8c
    (cherry picked from commit b291e502b27eaa5ffb4d5d971fe0f6aac8f3b866)

[Dougal Matthews (d0ugal)]

Seen failing in https://review.openstack.org/#/c/525926/ and https://review.openstack.org/#/c/528291/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_ssh_actions.SSHActionsTestsV2.test_run_ssh_proxied_action

Disabling patch incoming...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/528999

[Dougal Matthews (d0ugal)]

Seen failing in https://review.openstack.org/#/c/525926/ and https://review.openstack.org/#/c/528291/

mistral_tempest_tests.tests.scenario.engine.actions.v2.test_ssh_actions.SSHActionsTestsV2.test_run_ssh_proxied_action

Disabling patch incoming...

[Dougal Matthews (d0ugal)]

I opened a bug for this test failure too. https://bugs.launchpad.net/mistral/+bug/1738974

[Dougal Matthews (d0ugal)]

See failing in https://review.openstack.org/#/c/528999/

mistral_tempest_tests.tests.api.v2.test_event_triggers.EventTriggerTestsV2.test_get_nonexistent_event_trigger

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/530290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530290
Committed: https://git.openstack.org/cgit/openstack/mistral-tempest-plugin/commit/?id=63a0fe20f98e0cb8316beb81ca77249ffdda29c5
Submitter: Zuul
Branch: master

commit 63a0fe20f98e0cb8316beb81ca77249ffdda29c5
Author: Dougal Matthews <email address hidden>
Date: Tue Dec 19 08:55:27 2017 +0000

    Disable unstable tempest test_run_ssh_proxied_action test

    This disables test_run_ssh_proxied_action. Currently it causes the tests to
    fail frequently which makes them less useful and blocks patches incorrectly.

    Related-Bug: #1736685
    Change-Id: I9bd4cb8f828eca3e3227d8e52ef916191a36b45f

[Dougal Matthews (d0ugal)]

I think we can close this. I am no longer seeing stability and this bug doesn't have a clear path to fixing/closing