500 internal server error If you check "Select all" and click "Run OSTF tests"

[Anastasia Palkina]

"build_id": "2014-10-30_04-21-22",
"ostf_sha": "f47fd1d66a7255213ee075d5c11b8f111f922000",
"build_number": "63",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "02c6bb2e54bbec76da33167eaf5f2e0b3e2e50a7",
"production": "docker",
"fuelmain_sha": "2ade7c571380a091048d103a6affff634b5b2520",
"astute_sha": "97eea90efe0a1f17b4934919d6e459d270c10372",
"feature_groups": ["mirantis", "techpreview"],
"release": "6.0-techpreview",
"release_versions": {"2014.2-6.0-techpreview": {"VERSION": {"build_id": "2014-10-30_04-21-22", "ostf_sha": "f47fd1d66a7255213ee075d5c11b8f111f922000", "build_number": "63", "api": "1.0", "nailgun_sha": "02c6bb2e54bbec76da33167eaf5f2e0b3e2e50a7", "production": "docker", "fuelmain_sha": "2ade7c571380a091048d103a6affff634b5b2520", "astute_sha": "97eea90efe0a1f17b4934919d6e459d270c10372", "feature_groups": ["mirantis", "techpreview"], "release": "6.0-techpreview", "fuellib_sha": "45b6fc42091a0a33d3e48fbe78b782ce743aedc1"}}},
"fuellib_sha": "45b6fc42091a0a33d3e48fbe78b782ce743aedc1"

1. Create new environment (CentOS, simple mode)
2. Choose neutron, vlan
3. Choose both Ceph
4. Add 1 controller, 1 compute+ceph, 2 ceph
5. Start deployment. It was succcessful
6. Open HealthCheck tab
7. Click "Select all" and click button "Run tests"
8. Server error occured (see screen)

[Sebastian Kalinowski (prmtl)]

Need a reproducer that will provide diagnostic snapshot.

[Anastasia Palkina (apalkina)]

I reproduced it on ISO #75

"build_id": "2014-11-04_16-38-46", "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa", "build_number": "75", "auth_required": true, "api": "1.0", "nailgun_sha": "35946b1f225c984f11915ba8e985584160f0b129", "production": "docker", "fuelmain_sha": "d498d9153494b412cc75900ab8a1f4e18bc26c13", "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-04_16-38-46", "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa", "build_number": "75", "api": "1.0", "nailgun_sha": "35946b1f225c984f11915ba8e985584160f0b129", "production": "docker", "fuelmain_sha": "d498d9153494b412cc75900ab8a1f4e18bc26c13", "astute_sha": "c72dac7b31646fbedbfc56a2a87676c6d5713fcf", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2a314f9d14ea045b4b917d01d6c8f9a732ca1d7f"}}}, "fuellib_sha": "2a314f9d14ea045b4b917d01d6c8f9a732ca1d7f"

[Sebastian Kalinowski (prmtl)]

SCF, moving do 6.1.

[Mike Scherbakov (mihgen)]

Assuming that this is consistent and repeatable behavior for this configuration, moving this to High priority issue for 6.0.
HealthCheck is very important feature for giving the health status of the environment. If it fails to work, then it is a serios problem to address.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134470
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=b21dc974e8edd43b3cd589cea35e8560fbdcce89
Submitter: Jenkins
Branch: master

commit b21dc974e8edd43b3cd589cea35e8560fbdcce89
Author: Sebastian Kalinowski <email address hidden>
Date: Thu Nov 13 11:08:03 2014 +0100

    Populate test cache if it's empty

    In commit e232d498a4c5fd8fbf4ca9c17f528ac6274146b3 bug (#1367359)
    when there were no TestRuns for cluster when OSTF tests were started
    without asking for a lost of available TestRuns.

    In the same case when tests cache is empty not all required
    objects (ClusterTestingPattern) will be created in DB that are
    needed to correctly run tests.

    Now we force cache repopulation if it's empty.

    Additionally removed some dead code that is a duplicate
    of modified one in this changestet.

    Change-Id: If7dd352f2a48c27b7aa9f41c6c5a4528021a8208
    Closes-Bug: #1387733

[Anastasia Palkina (apalkina)]

Cannot reproduce on latest ISOs

"build_id": "2014-11-22_22-01-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "129", "auth_required": true, "api": "1.0", "nailgun_sha": "7196c478bfe0b5cff97077c1829009b9dbc3ee92", "production": "docker", "fuelmain_sha": "0dd338b6280843b4c45b46635528a6c43939c772", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-22_22-01-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "129", "api": "1.0", "nailgun_sha": "7196c478bfe0b5cff97077c1829009b9dbc3ee92", "production": "docker", "fuelmain_sha": "0dd338b6280843b4c45b46635528a6c43939c772", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2bfa9431a4839efae9c75d6b133df24c0f11c868"}}}, "fuellib_sha": "2bfa9431a4839efae9c75d6b133df24c0f11c868"

[Anastasia Palkina (apalkina)]

Reproduced on ISO #4 for 6.0

"build_id": "2014-11-24_22-41-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "4", "auth_required": true, "api": "1.0", "nailgun_sha": "603a8d438dc7a3cf6286eb9f16deb8137f47d703", "production": "docker", "fuelmain_sha": "45b21f7bdb061b59b80f8d126d9a6f6e50505a0d", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-24_22-41-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "4", "api": "1.0", "nailgun_sha": "603a8d438dc7a3cf6286eb9f16deb8137f47d703", "production": "docker", "fuelmain_sha": "45b21f7bdb061b59b80f8d126d9a6f6e50505a0d", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "893883f7fa8ffc5dde975b6806e538a11969a15b"}}}, "fuellib_sha": "893883f7fa8ffc5dde975b6806e538a11969a15b"

[Dmitry Pyzhov (dpyzhov)]

Can be affected by parallel deployment

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/138387

[Kamil Sambor (ksambor)]

I'm still checking possibilities of leaking of sessions between requests but maybe someone else will find solution of this problem.

[Dima Shulyak (dshulyak)]

In my understanding - if there is any problems with shared sessions between requests, this failure will be much more repeatable.
I am trying to reproduce it for second day, by running tests for 2 clusters in different order, and the only way i was able to break it is by next steps:

1. Go to heltcheck tab
2. Restart ostf
3. Run tests (there is some stale data on UI)
4. ostf failed, but after page refresh all works well

I am lowering priority and moving this bug to 6.1

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kamil Sambor (<email address hidden>) on branch: master
Review: https://review.openstack.org/138387

[Dmitry Pyzhov (dpyzhov)]

Please reopen this bug if you reproduce it. Looks like original fix works fine.

[Tatyanka (tatyana-leontovich)]

verified 525