Dell Storage Center driver API calls temporary failure

[Sean McGinnis]

Something in one of our lab environments causes connectivity problems between the driver and the management server. ConnectionError exceptions are observed in the tempest log files with the error code that there is no route to host. But immediately prior to that error and immediately after there are successful API calls to that host.

Retries are needed on these connection errors to handle the temporary failure.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155479
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=49dd3785229200d649d1291e556ae30566763bce
Submitter: Jenkins
Branch: master

commit 49dd3785229200d649d1291e556ae30566763bce
Author: Sean McGinnis <email address hidden>
Date: Thu Feb 12 14:52:56 2015 -0600

    Dell Storage Center: Add retries to API calls

    In heavily loaded networks we have seen some cases
    of temporary ConnectionErrors when making REST API
    calls. There are usually successful calls just prior
    and immediately after these failures, so it appears
    to be a transient condition.

    This patch utilizes the recently merged retry decorator
    to add some retry handling to the REST API calls when
    this condition is encountered.

    With a test script looping through repeatedly running
    CI against this first patch I was able to validate
    that it addresses the issue. Out of 20 runs, a full
    2/3 of the tests passed showing that the retry was
    used. These test runs would have failed without the
    retry.

    The output from these test runs can be viewed here:

    http://oslogs.compellent.com/?C=N;O=D

    Pertinent results are dell-sc-iscsi-1554792015-12*

    Once merged we should be able to enable full third
    party CI testing with some expectation of reliable
    results.

    Closes-Bug: 1422742
    Change-Id: I1279a7696068aee534498bb18f4f6b4fbb7a33d5