Adding data to Redis cluster fails in gate

[Peter Stachowski]

It seems as if the fix for https://bugs.launchpad.net/trove/+bug/1645775 didn't work. Adding a new bug now to track the issue.

I believe it might be due to the fact that some machines are way faster now (including the test devstack machine), and the data is being added before the cluster is actually ready to be used. A possible solution is to try multiple times to add data, as is done with the verify part of the tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415516

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416165

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416361

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415516
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=42c7a40d43bc83b56786eb63160791acb651389d
Submitter: Jenkins
Branch: master

commit 42c7a40d43bc83b56786eb63160791acb651389d
Author: Peter Stachowski <email address hidden>
Date: Wed Dec 28 09:09:31 2016 -0800

    Move try-again loop around all data methods

    The verify-data method will try multiple times to
    see if the data is on the node in question. This
    functionality has been moved down the stack so that
    all data methods (add, verify, remove) will attempt
    multiple times.

    Change-Id: Ic3ffce6cdcc1e7d98d484c30c65b6dc01535873a
    Partial-Bug: 1652964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416165
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=606c59737fe29d74be89beee07ecb67f0bb98519
Submitter: Jenkins
Branch: master

commit 606c59737fe29d74be89beee07ecb67f0bb98519
Author: Peter Stachowski <email address hidden>
Date: Mon Jan 2 23:30:40 2017 -0800

    Have scenario tests retrive guest log on error

    As an aid to debug guest failures, a Metaclass was added
    to the TestRunner class that allows failed tests to
    pull back and echo the guest log from 'registered'
    instances.

    This uses the guest-log feature that is available for
    all datastores.

    IDs for instances created in the tests were registered
    to report on failures.

    Two guest-log tests were also commented out, as they
    seem to not work properly: see
    https://bugs.launchpad.net/trove/+bug/1653614
    This was discovered while testing the new retrieval
    code. Other tests were also modified so that
    'SkipTest' exceptions would be raised properly.

    Change-Id: I448bd2f0181351ef1536e20c41f9d45f95478587
    Partial-Bug: 1652964

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416361
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=55d8598d206cdfbc7cc7c07fdf9ba2f5fa59e81d
Submitter: Jenkins
Branch: master

commit 55d8598d206cdfbc7cc7c07fdf9ba2f5fa59e81d
Author: Peter Stachowski <email address hidden>
Date: Tue Jan 3 13:57:06 2017 -0800

    Install Redis 3.2.6 by compilation

    The version of Redis installed by the current elements
    uses a ppa that is stalled at 3.0.7. This seems to
    have some issues that are fixed in later versions.

    The recommended method of installing Redis is to compile
    the code yourself, so the elements have been changed
    to do this.

    The version was bumped to 3.2.6 (latest stable) however
    the Redis cluster tests were not reenabled as there still
    seems to be an issue with them.

    New config options were added to the template; these
    may need to be updated to make sure all the new ones
    are there.

    Change-Id: I1a604b15ade815cdb51c5484d4285e504508a6c4
    Partial-Bug: 1652964