Need increase timeout in mysql check in test mysql termination

[Tatyanka]

Test https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.ha_neutron_destructive/83/testReport/(root)/ha_neutron_mysql_termination/ha_neutron_mysql_termination/

 failed by timeout on step waiting while mysqld is running for node 2, but actually seems we do not wait enough(see explanation below):

we sent pkill -9 -x "mysqld" command near 04:15 :48 at 04:16 pacemaker report that unexpected error happens with p_mysql resource and start ocf script to recover it at 04:20:36, at the same time we report that mysql is not run on node 2 and interrupt the test

Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql']/lrm_rsc_op[@id='p_mysql_last_0']: @operation_key=p_mysql_start_0, @operation=start, @transition-key=154:321:0:88459649-6a7a-4fc6-a905-dabdfec48219, @transition-magic=2:1;154:321:0:88459649-6a7a-4fc6-a905-dabdfec48219, @call-id=300, @rc-code=1, @op-status=2, @exec-time=300001
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql']/lrm_rsc_op[@id='p_mysql_last_failure_0']: @operation_key=p_mysql_start_0, @operation=start, @transition-key=154:321:0:88459649-6a7a-4fc6-a905-dabdfec48219, @transition-magic=2:1;154:321:0:88459649-6a7a-4fc6-a905-dabdfec48219, @call-id=300, @op-status=2, @interval=0, @last-rc-change=1441685736, @exec-time=300001, @op-digest=e930775
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=node-1.test.domain.local/crmd/698, version=0.150.26)
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/239)
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/241)
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_perform_op: Diff: --- 0.150.26 2
Sep 08 04:20:36 [16815] node-2.test.domain.local cib: info: cib_perform_op: Diff: +++ 0.150.27 (null)

Logs from system test about resolution of mysql
2015-09-08 04:15:48,052 - INFO test_failover_base.py:304 -- Terminating MySQL on slave-02
2015-09-08 04:20:48,078 - ERROR checkers.py:435 -- MySQL daemon is down on slave-02

[Dmitry Tyzhnenko (dtyzhnenko)]

Last 4 runs of swarm test was passed, so I decrease importance.

[Dmitry Tyzhnenko (dtyzhnenko)]

https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/13/testReport/(root)/ha_neutron_mysql_termination/
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/12/testReport/(root)/ha_neutron_mysql_termination/
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/11/testReport/(root)/ha_neutron_mysql_termination/
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/9/testReport/(root)/ha_neutron_mysql_termination/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234368

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234368
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=fa0b557bfd4401bf991213db856723fdb85cd629
Submitter: Jenkins
Branch: master

commit fa0b557bfd4401bf991213db856723fdb85cd629
Author: Maksym Strukov <email address hidden>
Date: Tue Oct 13 20:51:06 2015 +0300

    Give additional time for pacemaker to recover mysql after kill

    Change-Id: I00a2f67cbe9c60d2ad24b6cc1c98e52d99d723dd
    Closes-Bug: #1493273