[OSTF]After succesfull cleanup button 'run test' does not appear

[Tatyanka]

{"build_id": "2014-05-12_01-10-31", "mirantis": "yes", "build_number": "193", "ostf_sha": "cdb075090b752246a9c43db3e918c42f645b5873", "nailgun_sha": "4477ba3a6efc4379a6509386e7a9e2e6ae832041", "production": "docker", "api": "1.0", "fuelmain_sha": "97d7f6d5461db3afc27f58160cf9f6985230d255", "astute_sha": "5813d9b537ba6ac95f668321c682f339aac57e05", "release": "5.0", "fuellib_sha": "112695da3558df7b8cdc1bd4d7bb214bed38bb78"}

Steps to reproduce:
1. Deploy cluster with savanna/murano on netroun
2. Run ostf
3. Stop the tests

Expected:
Clean up starts, as soon as it finishes - button run tests available

Actual:
Cleanup starts and finishes with success, but button run tests is not available.
When I run stop test one more time I see, that current status of fuel_health is stopped:

Put on http://10.108.0.2:8000/ostf/test_runs
Host: 10.108.0.2:8000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://10.108.0.2:8000/
Content-Length: 29
Cookie: i18next=en-US
Connection: keep-alive

Response:
500 internal error

I see in the log that ostf proccess was killed and test was stopped

[Vitaly Kramskikh (vkramskikh)]

Seems to be a backend bug. Sometimes tests fail to stop and current testrun hangs in "running" state. Attempting to stop the tests for a second time leads to 500 error

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93587
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=353f918197ec53a00127fd28b9151f248a2a2d30
Submitter: Jenkins
Branch: master

commit 353f918197ec53a00127fd28b9151f248a2a2d30
Author: Artem Roma <email address hidden>
Date: Wed May 14 18:43:11 2014 +0300

    Fix bug with cleanup running for locked tests

    Code for lock aquiring modified.

    - locks are aquired inside try statement,
     prevents from being interrupted outside of main flow
    - prevent 500 error in case process was already stopped
    - remove SIGUSR1 interrupt handler with SIG_DFL
     which simply returns 0

    Change-Id: Icc31510db705b11c2b628604d0480d8da6aedeca
    Closes-Bug: #1318631

[Anastasia Palkina (apalkina)]

Verified on ISO #214
"build_id": "2014-05-21_01-10-31",
"mirantis": "yes",
"build_number": "214",
"ostf_sha": "353f918197ec53a00127fd28b9151f248a2a2d30",
"nailgun_sha": "0b6e8eabaccad2aa29519561ce7cde9df9292964",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "910f262f85e94bef08e0e9b9d6230ad890bf139e",
"astute_sha": "9a0d86918724c1153b5f70bdae008dea8572fd3e",
"release": "5.0",
"fuellib_sha": "3d92142a5643af82596f0450e39282550a45e5db"