In system tests we check HA for RabbitMQ cluster by suspending controllers and verifying it passes health checks, for example:
https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/tests/tests_strength/test_failover_base.py#L1043
Then we resume suspended node and perform health checks once again:
https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/tests/tests_strength/test_failover_base.py#L1079
This scenario is not applicable for real hardware environments and could cause additional issues for RabbitMQ recovering script, so recovering takes more time than usually:
Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/helpers/decorators.py", line 80, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover.py", line 313, in ha_neutron_test_3_1_rabbit_failover
super(self.__class__, self).test_3_1_rabbit_failover()
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover_base.py", line 1025, in test_3_1_rabbit_failover
self.fuel_web.assert_ha_services_ready(cluster_id, timeout=300)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/__init__.py", line 58, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 155, in assert_ha_services_ready
interval=20, timeout=timeout)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 108, in _wait
return raising_predicate()
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 154, in <lambda>
should_fail=should_fail),
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/__init__.py", line 58, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 1141, in run_ostf
failed_test_name=failed_test_name, test_sets=test_sets)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/__init__.py", line 58, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 258, in assert_ostf_run
indent=1)))
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
raise ASSERTION_ERROR(message)
AssertionError: Failed 2 OSTF tests; should fail 0 tests. Names of failed tests: [
{
"RabbitMQ availability (failure)": "Time limit exceeded while waiting for to finish. Please refer to OpenStack logs for more details."
},
{
"RabbitMQ replication (failure)": "Failed to establish AMQP connection to 5673/tcp port on 10.109.7.4 from controller node! Please refer to OpenStack logs for more details."
}]
We need to replace suspend/resume actions by destroy/start in system tests for RabbitMQ failover.
Fix proposed to branch: master
Review: https:/