Fix Galera-based cluster tests running in neutron gat

[Peter Stachowski]

With the switch to neutron in the gate, Galera-based cluster tests (i.e. PXC, MariaDB) started to fail regularly and were disabled (at least PXC was, as MariaDB is still non-voting). The error is:

    2016-11-25 16:36:44.396479 | 2016-11-25 16:36:44.396 | ERROR: proboscis.case.MethodTest (cluster_create)
    2016-11-25 16:36:44.397838 | 2016-11-25 16:36:44.397 | ----------------------------------------------------------------------
    2016-11-25 16:36:44.399540 | 2016-11-25 16:36:44.399 | Traceback (most recent call last):
    2016-11-25 16:36:44.402994 | 2016-11-25 16:36:44.402 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    2016-11-25 16:36:44.404376 | 2016-11-25 16:36:44.404 | compatability.capture_type_error(s_func)
    2016-11-25 16:36:44.406318 | 2016-11-25 16:36:44.405 | File "/usr/local/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    2016-11-25 16:36:44.409304 | 2016-11-25 16:36:44.409 | func()
    2016-11-25 16:36:44.411420 | 2016-11-25 16:36:44.411 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    2016-11-25 16:36:44.412751 | 2016-11-25 16:36:44.412 | func(test_case.state.get_state())
    2016-11-25 16:36:44.414350 | 2016-11-25 16:36:44.413 | File "/opt/stack/new/trove/trove/tests/scenario/groups/cluster_actions_group.py", line 51, in cluster_create
    2016-11-25 16:36:44.415927 | 2016-11-25 16:36:44.415 | self.test_runner.run_cluster_create()
    2016-11-25 16:36:44.417736 | 2016-11-25 16:36:44.417 | File "/opt/stack/new/trove/trove/tests/scenario/runners/cluster_actions_runners.py", line 77, in run_cluster_create
    2016-11-25 16:36:44.420361 | 2016-11-25 16:36:44.420 | expected_task_name, expected_instance_states, expected_http_code)
    2016-11-25 16:36:44.422437 | 2016-11-25 16:36:44.421 | File "/opt/stack/new/trove/trove/tests/scenario/runners/cluster_actions_runners.py", line 102, in assert_cluster_create
    2016-11-25 16:36:44.423794 | 2016-11-25 16:36:44.423 | cluster_instances, expected_instance_states)
    2016-11-25 16:36:44.426114 | 2016-11-25 16:36:44.425 | File "/opt/stack/new/trove/trove/tests/scenario/runners/test_runners.py", line 466, in assert_all_instance_states
    2016-11-25 16:36:44.427378 | 2016-11-25 16:36:44.427 | self.fail(str(task.poll_exception()))
    2016-11-25 16:36:44.428716 | 2016-11-25 16:36:44.428 | File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 87, in poll_exception
    2016-11-25 16:36:44.430651 | 2016-11-25 16:36:44.429 | return self.wait()
    2016-11-25 16:36:44.431858 | 2016-11-25 16:36:44.431 | File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
    2016-11-25 16:36:44.433101 | 2016-11-25 16:36:44.432 | current.throw(*self._exc)
    2016-11-25 16:36:44.434854 | 2016-11-25 16:36:44.434 | File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 136, in _run_loop
    2016-11-25 16:36:44.436319 | 2016-11-25 16:36:44.436 | result = func(*self.args, **self.kw)
    2016-11-25 16:36:44.437661 | 2016-11-25 16:36:44.437 | File "/opt/stack/new/trove/trove/common/utils.py", line 196, in poll_and_check
    2016-11-25 16:36:44.439376 | 2016-11-25 16:36:44.439 | obj = retriever()
    2016-11-25 16:36:44.441125 | 2016-11-25 16:36:44.440 | File "/opt/stack/new/trove/trove/tests/scenario/runners/test_runners.py", line 450, in <lambda>
    2016-11-25 16:36:44.443397 | 2016-11-25 16:36:44.443 | require_all_states=require_all_states)
    2016-11-25 16:36:44.444788 | 2016-11-25 16:36:44.444 | File "/opt/stack/new/trove/trove/tests/scenario/runners/test_runners.py", line 496, in _assert_instance_states
    2016-11-25 16:36:44.446711 | 2016-11-25 16:36:44.446 | time_out=self.def_timeout)
    2016-11-25 16:36:44.448211 | 2016-11-25 16:36:44.447 | File "/opt/stack/new/trove/trove/common/utils.py", line 216, in poll_until
    2016-11-25 16:36:44.449641 | 2016-11-25 16:36:44.449 | sleep_time=sleep_time, time_out=time_out).wait()
    2016-11-25 16:36:44.451617 | 2016-11-25 16:36:44.451 | File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
    2016-11-25 16:36:44.453096 | 2016-11-25 16:36:44.452 | return hubs.get_hub().switch()
    2016-11-25 16:36:44.455086 | 2016-11-25 16:36:44.454 | File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
    2016-11-25 16:36:44.456471 | 2016-11-25 16:36:44.456 | return self.greenlet.switch()
    2016-11-25 16:36:44.457840 | 2016-11-25 16:36:44.457 | File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 136, in _run_loop
    2016-11-25 16:36:44.459335 | 2016-11-25 16:36:44.458 | result = func(*self.args, **self.kw)
    2016-11-25 16:36:44.460880 | 2016-11-25 16:36:44.460 | File "/opt/stack/new/trove/trove/common/utils.py", line 196, in poll_and_check
    2016-11-25 16:36:44.462213 | 2016-11-25 16:36:44.461 | obj = retriever()
    2016-11-25 16:36:44.463589 | 2016-11-25 16:36:44.463 | File "/opt/stack/new/trove/trove/tests/scenario/runners/test_runners.py", line 494, in <lambda>
    2016-11-25 16:36:44.464963 | 2016-11-25 16:36:44.464 | fast_fail_status=fast_fail_status),
    2016-11-25 16:36:44.467508 | 2016-11-25 16:36:44.467 | File "/opt/stack/new/trove/trove/tests/scenario/runners/test_runners.py", line 581, in _has_status
    2016-11-25 16:36:44.469870 | 2016-11-25 16:36:44.468 | % (instance_id, instance.status))
    2016-11-25 16:36:44.472961 | 2016-11-25 16:36:44.472 | RuntimeError: Instance '<Instance cluster_id=974546da-c864-434e-b59a-17ed305b1c56, created=2016-11-25T16:27:11, datastore={u'version': u'5.6', u'type': u'pxc'}, flavor={u'id': u'15', u'links': [{u'href': u'https://127.0.0.1:8779/v1.0/a10363db04f946689718bea29cc78f73/flavors/15', u'rel': u'self'}, {u'href': u'https://127.0.0.1:8779/flavors/15', u'rel': u'bookmark'}]}, id=01f8380c-65e4-4b4b-88a4-a8de0c0ae6e2, links=[{u'href': u'https://127.0.0.1:8779/v1.0/a10363db04f946689718bea29cc78f73/instances/01f8380c-65e4-4b4b-88a4-a8de0c0ae6e2', u'rel': u'self'}, {u'href': u'https://127.0.0.1:8779/instances/01f8380c-65e4-4b4b-88a4-a8de0c0ae6e2', u'rel': u'bookmark'}], name=test_cluster-member-1, region=RegionOne, status=BUILD, updated=2016-11-25T16:27:11, volume={u'size': 1}>' acquired a fast-fail status: ERROR

This will need to be debugged (in the gate probably) and a fix submitted.

[Peter Stachowski (peterstac)]

This may be related to the failure documented in https://bugs.launchpad.net/trove/+bug/1645775