bvt deployment waits too long to fail

[Aleksandra Fedorova]

http://jenkins-product.srt.mirantis.net:8080/job/9.0.custom.ubuntu.bvt_2/416/

In this deployment see diagnostic snapshot attached according to astute.log there were three broken sets of tasks (HA_proxy, mysql and keystone_admin smth), which were hanging in 'running' state for more then half an hour, without any changes.

We should catch such situations early and fail the build.

[Kyrylo Galanov (kgalanov)]

Hi Aleksandra,

There are 3 levels of timeouts:
1. puppet resource provider timeout
2. astute task timeout
3. automated test timeout

I cannot shrink timeouts inspired by intuition because it might break large scale deployments.
Which timeout specifically do you propose to decrease?

Best regards,
Kyrylo

[Aleksandra Fedorova (bookwar)]

Do we need to rely on timeouts here?

Can't we recognize that deployment got into the loop without any changes in task statuses and fail based on that?

[Kyrylo Galanov (kgalanov)]

Aleksandra,

If you can provide criteria how we can unambiguously determine that the deployment has failed before timeout is met, I think it is possible to implement that in code.
My concern is that decreasing timeouts can _possibly_ impact the deployments which need more time to complete.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/320281

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/320281
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=bf14da9bf265638ed661abc3557b0c41f9f42bc5
Submitter: Jenkins
Branch: master

commit bf14da9bf265638ed661abc3557b0c41f9f42bc5
Author: Kyrylo Galanov <email address hidden>
Date: Tue May 24 09:57:01 2016 +0200

    Decrease task timeouts according to actual duration

    Most of tasks had timeout equal to 3600 seconds whereas actual duration
    is no more than 1 minute.
    Minimal timeout is 60 seconds even if task is complete in a few seconds.
    New timeout ~ duration * 2.5

    Change-Id: Iea9ee8f5038f5fcfd9dcdfc2d9ba964eab035549
    Closes-bug: #1558505