Timeout of web test on fuel-jenkins

[Dmitry Pyzhov]

Lots of failed checks because of timeot. For example: http://fuel-jenkins.vm.mirantis.net:8080/job/stackforge-verify-fuel-web/3397/console

You can compare it with passed test: http://fuel-jenkins.vm.mirantis.net:8080/job/stackforge-verify-fuel-web/3398/console

....
PASS Number of ready nodes is correct
PASS Reset button is disabled after successful reset
PASS 51 tests executed, 51 passed, 0 failed.
<----------- According to logs it stuck right here ------------->
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade None -> 1a1504d469f8, fuel_5_0
/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/default.py:475: SAWarning: Unicode type received non-unicode bind param value.
  processors[key](compiled_params[key])
Test file: /home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/ui_tests/test_node_disk.js
# Testing nodes disks
PASS Node details popup was opened
PASS Node disks configuration screen appears
....

[Dmitry Pyzhov (dpyzhov)]

We have a deadlock between this two commands:
https://github.com/stackforge/fuel-web/blob/8185bba4f72c5bee4e2d2cd780d4c917a95507aa/run_tests.sh#L451-L456
https://github.com/stackforge/fuel-web/blob/8185bba4f72c5bee4e2d2cd780d4c917a95507aa/run_tests.sh#L248

[Dmitry Pyzhov (dpyzhov)]

2014-05-08 02:56:08 PDT ERROR: deadlock detected
2014-05-08 02:56:08 PDT DETAIL: Process 15548 waits for AccessExclusiveLock on relation 18495393 of database 16527; blocked by process 15277.
        Process 15277 waits for AccessShareLock on relation 18495321 of database 16527; blocked by process 15548.
        Process 15548: DROP TABLE IF EXISTS clusters CASCADE
        Process 15277: SELECT nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.cluster_id AS nodes_cluster_id, nodes.name AS nodes_name, nodes.status AS nodes_status, nodes.meta AS nodes_meta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.fqdn AS nodes_fqdn, nodes.manufacturer AS nodes_manufacturer, nodes.platform_name AS nodes_platform_name, nodes.kernel_params AS nodes_kernel_params, nodes.progress AS nodes_progress, nodes.os_platform AS nodes_os_platform, nodes.pending_addition AS nodes_pending_addition, nodes.pending_deletion AS nodes_pending_deletion, nodes.error_type AS nodes_error_type, nodes.error_msg AS nodes_error_msg, nodes.timestamp AS nodes_timestamp, nodes.online AS nodes_online, nodes.agent_checksum AS nodes_agent_checksum
        FROM nodes
        WHERE nodes.id = '1'
2014-05-08 02:56:08 PDT HINT: See server log for query details.
2014-05-08 02:56:08 PDT STATEMENT: DROP TABLE IF EXISTS clusters CASCADE

[Dmitry Pyzhov (dpyzhov)]

We should either get rid of deadlock or shutdown server before each dropdb run.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/92830

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92830
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=e07c52a81727af9988a57bfe2987eecda8f18a67
Submitter: Jenkins
Branch: master

commit e07c52a81727af9988a57bfe2987eecda8f18a67
Author: Aleksey Kasatkin <email address hidden>
Date: Thu May 8 16:01:51 2014 +0300

    Run DROPDB when server is not running

    This fix covers 90% (20 out of 22) deadlocks have been seen in test logs.
    One case was caused by 'dropdb' in cli tests themselves.
    We should get rid of 'dropdb' in cli tests to avoid these deadlocks.
    One another case doesn't seem to be connected with this issue.
    test_task_managers.test_deletion_during_deployment test refactored.

    Closes-Bug: #1317432

    Change-Id: Id11785103db392d41b0d0c35223fe4476ac8e85f