Comment 16 for bug 1621798

Today we have manually executed Tempest on 4 different OpenStack configurations, the results are good [1][2][3][4] (summary pass rate is 98.5%). CI jobs are still broken for the moment and we are working on the fixes.

We also applied several fixes in Tempest and QA scripts to fix the issue on CI:
1) We removed mongo database from the controller nodes
2) We added default configurations with 1 controller to check if the issue reproduced on environment with one controller
3) We added the custom change in Tempest [5] to get the detailed information about the errors with 401 status code (the root of the issue with Tempest results degradation). We expect that this change will help to understand what is going on with keystone during the tests.
4) We removed parallel execution of Tempest tests from CI jobs and now we run all tests in one thread to avoid any raise condition issues.
5) We added new servers to run different deployments / tests with different fixes in parallel.

Today we also merged the fix to Keystone [6] which fixes another Tempest-related issue [7] and we expect that the fix for the issue can also help to fix low Tempest pass rate.

Short Summary:
We still have an issue with Tempest on CI, the root of the issue is unknown yet but the issue which was fixed today in Keystone can probably affect it, we are waiting for the nightly jobs to check it.
We can't reproduce it manually, this is why we think it is not a blocker issue for the product, OpenStack components work fine and we just need to finally debug the issue to make sure that we don't have any hard-reproducible critical defects in the product.

We will continue to run Tempest test suite manually even on the weekends until we fix the issue with CI jobs.

[1] https://mirantis.testrail.com/index.php?/runs/view/20833&group_by=cases:section_id&group_order=asc (pass rate: 99%)
[2] https://mirantis.testrail.com/index.php?/runs/view/20835&group_by=cases:section_id&group_order=asc (pass rate: 99%)
[3] https://mirantis.testrail.com/index.php?/runs/view/20870&group_by=cases:section_id&group_order=asc (pass rate: 99%)
[4] https://mirantis.testrail.com/index.php?/runs/view/20874&group_by=cases:section_id&group_order=asc (pass rate: 98%)
[5] https://review.openstack.org/#/c/371418/
[6] https://review.fuel-infra.org/#/c/26523/
[7] https://bugs.launchpad.net/mos/+bug/1582220