[system-tests] Test destroy ha controller on ubuntu neutron failed on ostf

[Tatyanka]

http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/46/consoleFull

Failed tests, fails: 2 should fail: 1 failed tests name: [{u'RabbitMQ availability (failure)': u'Cannot retrieve cluster nodes list for 10.108.22.6 controller. Please refer to OpenStack logs for more details.'}, {u'Check that required services are running (failure)': u'Some nova services have not been started.. Please refer to OpenStack logs for more details.'}]

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/helpers/decorators.py", line 51, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover.py", line 142, in ha_destroy_controllers
    should_fail=1)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 554, in run_ostf
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/models/fuel_web_client.py", line 194, in assert_ostf_run
    failed_tests_res))
  File "/usr/lib/python2.7/dist-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Failed tests, fails: 2 should fail: 1 failed tests name: [{u'RabbitMQ availability (failure)': u'Cannot retrieve cluster nodes list for 10.108.22.6 controller. Please refer to OpenStack logs for more details.'}, {u'Check that required services are running (failure)': u'Some nova services have not been started.. Please refer to OpenStack logs for more details.'}]

In OSTF logs:
paramiko.transport: DEBUG: [chan 1] EOF received (1)
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 186, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/common/amqp_client.py", line 43, in list_nodes
    output = self.ssh.exec_command("rabbitmqctl cluster_status")
  File "/usr/lib/python2.6/site-packages/fuel_health/common/ssh.py", line 168, in exec_command
    strerror=''.join(err_data).join(out_data))
SSHExecCommandFailed: Command 'rabbitmqctl cluster_status', exit status: 2, Error:
Cluster status of node 'rabbit@node-4' ...
Error: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}}

paramiko.transport: DEBUG: EOF in transport thread

[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|--------|---------------------|---------|-------------|-------------------|------------|---------------|-------
3 | ready | slave-03_controller | 1 | 10.108.22.5 | 4a:96:9c:2f:91:47 | controller | | True
2 | ready | slave-04_compute | 1 | 10.108.22.4 | 6e:52:53:14:0f:4d | compute | | True
5 | ready | slave-05_compute | 1 | 10.108.22.7 | 06:f9:8f:1e:f1:4d | compute | | True
4 | ready | slave-01_controller | 1 | 10.108.22.6 | 26:28:e6:03:f9:40 | controller | | True
1 | ready | slave-02_controller | 1 | 10.108.22.3 | 8a:48:e6:cb:10:48 | controller | | False

so offline node was 10.108.22.3 and ostf test shoul pass

2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', '<email address hidden>', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', '<email address hidden>'] server compress:['none', '<email address hidden>'] client lang:[''] server lang:[''] kex follows?False
paramiko.transport: DEBUG: Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
paramiko.transport: DEBUG: using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
paramiko.transport: DEBUG: Switch to new keys ...
paramiko.transport: DEBUG: Adding ssh-rsa host key for 10.108.22.6: 52274727c2578aefa4514fadfe6be6a9
paramiko.transport: DEBUG: Trying key ac1b8062fa3a4861300af82caeb9e730 from /root/.ssh/id_rsa
paramiko.transport: DEBUG: userauth is OK
paramiko.transport: INFO: Authentication (publickey) successful!
paramiko.transport: DEBUG: [chan 1] Max packet in: 34816 bytes
paramiko.transport: DEBUG: [chan 1] Max packet out: 32768 bytes
paramiko.transport: INFO: Secsh channel 1 opened.
paramiko.transport: DEBUG: [chan 1] Sesch channel 1 request ok
paramiko.transport: DEBUG: [chan 1] Sesch channel 1 request ok
paramiko.transport: DEBUG: [chan 1] EOF sent (1)
paramiko.transport: DEBUG: [chan 1] EOF received (1)
paramiko.transport: DEBUG: Dropping user packet because connection is dead.

[Ryan Moe (rmoe)]

Won't fix for 5.1.2 and 6.0.1 as medium priority bugs aren't backported to stable releases.

[Tatyanka (tatyana-leontovich)]

http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.centos.thread_5/99/consoleFull

I reverted prepare nova snapshot and execute steps manually:
1. destroy one controller
2. check pacemaker status as in tests
3. Check that galera is ok
4. run ostf ha (wait while it pass)
5. run ostf sanity
6. run ostf smoke
and in this order cluster succesfully recovered
than I do rever again (btw we have active nodes at this moment :) And would be better to pause them before revert)
and then as in tests I do next actions:
1. destroy one controller
2. check pacemaker status as in tests
3. Check that galera is ok
4. run ostf ha (ha,smoke,sanity) and fail the same as in test, because keystone and memchached need so time for recovery after revert of snapshot
I wait for 3 minutes, run ostf again and all the ostf tests are passed, so we need to run ostf as in first variant

[Tatyanka (tatyana-leontovich)]

move to invalid for 7.0 according to issue is not reproduced more then month