trove-integration test are failing due to timeout

[Dan Nguyen]

The following tests are failing due to lower timeouts.

AfterConfigurationsCreation
 test_assign_configuration_to_invalid_instance OK 0.03
 test_assign_configuration_to_valid_instance OK 0.10
 test_assign_configuration_to_instance_with_config OK 0.09
 test_get_configuration_details_from_instance_validation ERROR

======================================================================
ERROR: proboscis.case.MethodTest (test_get_configuration_details_from_instance_validation)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/local/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/usr/local/lib/python2.7/dist-packages/proboscis/decorators.py", line 59, in new_method
    return func(*kargs, **kwargs)
  File "/opt/stack/trove/trove/tests/api/configurations.py", line 278, in test_get_configuration_details_from_instance_validation
    configuration_id)
  File "/opt/stack/trove/trove/tests/api/configurations.py", line 77, in _test_configuration_is_applied_to_instance
    instance_test = instance_info.dbaas.instances.get(instance.id)
  File "/opt/stack/python-troveclient/troveclient/v1/instances.py", line 116, in get
    "instance")
  File "/opt/stack/python-troveclient/troveclient/base.py", line 165, in _get
    resp, body = self.api.client.get(url)
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 237, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 231, in _cs_request
    return request()
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 221, in request
    **kwargs)
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 209, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 181, in request
    self.raise_error_from_status(resp, None)
  File "/opt/stack/python-troveclient/troveclient/compat/client.py", line 193, in raise_error_from_status
    raise exceptions.from_response(resp, body)
BadRequest: Bad request (HTTP Bad request (HTTP 400)

The fix for now is to increase the threshold in all the places where the test method is invoked.

 @time_out(10)
    def test_get_configuration_details_from_instance_validation(self):
        # validate that the configuraiton was applied correctly to the instance
        inst = instance_info.dbaas.instances.get(instance_info.id)
        configuration_id = inst.configuration['id']
        assert_not_equal(None, inst.configuration['id'])
        _test_configuration_is_applied_to_instance(instance_info,
                                                   configuration_id)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/97997

[Dan Nguyen (daniel-a-nguyen)]

For future reference here is the out put from logs that confirm the issue.

trove-api.log:2014-06-04 17:14:20.435 ERROR trove.guestagent.api [-] Timeout while waiting on RPC response - topic: "guestagent.323929a3-0217-4d18-8b88-f45ae8a92428", RPC method: "get_filesystem_stats" info: "<unknown>"

Output from running the test with --stop and --group=blackbox show this.

troveclient.compat.client: DEBUG: RESP:{'status': '400', 'content-length': 34, 'content-type': 'text/plain'} Time out after waiting 10 seconds.

Note, if you run the curl commands that resulted in a 400 response above you will see that it is indeed successfully (200).
This also supports an issue with timing.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97997
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=1ffe49cdbf809114b510487428b63aa648d1d704
Submitter: Jenkins
Branch: master

commit 1ffe49cdbf809114b510487428b63aa648d1d704
Author: daniel-a-nguyen <email address hidden>
Date: Wed Jun 4 17:46:47 2014 -0700

    Increase time_out to 30 sec for failing int-tests

    The test_get_configuration_details_from_instance_validation
    tests were randomly timing out during certain int-test runs.
    So the timeouts for these have been upped from 10s to 20s to
    allow them to succeed more consistently.

    Change-Id: I38451a2172397df9703e5fc90101d81c4fa953c4
    Closes-Bug: #1326591