Test 'ha_neutron_test_3_1_rabbit_failover' failed on 7.0.system_test.ubuntu.ha_neutron_destructive

[Ekaterina Shutova]

Test 'ha_neutron_test_3_1_rabbit_failover' failed with:
AssertionError: Failed 3 OSTF tests; should fail 0 tests. Names of failed tests: [
 {
  "Check pacemaker status (failure)": "On the controller node-4.test.domain.local, resource master_p_rabbitmq-server is active but failed to start (managed).. Please refer to OpenStack logs for more details."
 },
 {
  "RabbitMQ availability (failure)": "Cannot retrieve cluster nodes Please refer to OpenStack logs for more details."
 },
 {
  "RabbitMQ replication (failure)": "Failed to establish AMQP connection to 5673/tcp port on 10.129.2.9 from controller node! Please refer to OpenStack logs for more details."
 }
]

Release: 7.0
Area: system_test
Test: ubuntu.ha_neutron_destructive

Logs are provided:
https://paste.mirantis.net/show/1826/
Path to failed build:
https://patching-ci.infra.mirantis.net/view/All/job/7.0.full.runner/5/

[Dmitry Belyaninov (dbelyaninov)]

Reproduced:

https://mirantis.testrail.com/index.php?/tests/view/4044581

[Maciej Relewicz (rlu)]

related: https://launchpad.net/bugs/1567355

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307623

[Bogdan Dobrelya (bogdando)]

Related bug 1541471 it seems has the same RC

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307623
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d64be61ada27a200894d7fd08c1803079cf3aed2
Submitter: Jenkins
Branch: master

commit d64be61ada27a200894d7fd08c1803079cf3aed2
Author: Maciej Relewicz <email address hidden>
Date: Tue Apr 19 10:04:45 2016 +0200

    Stop process when rabbit is running but is not connected to master.

    It's should goes down due to avoid split brain.

    Change-Id: I4c51f8608702f2284d835ba9c3c9070b2c329ed8
    Closes-Bug: #1541471
    Upstream PR: https://github.com/rabbitmq/rabbitmq-server/pull/758

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309928
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=43886f7b51c4e67d5006f099c1b35efd2dc71685
Submitter: Jenkins
Branch: stable/mitaka

commit 43886f7b51c4e67d5006f099c1b35efd2dc71685
Author: Maciej Relewicz <email address hidden>
Date: Tue Apr 19 10:04:45 2016 +0200

    Stop process when rabbit is running but is not connected to master.

    It's should goes down due to avoid split brain.

    Change-Id: I4c51f8608702f2284d835ba9c3c9070b2c329ed8
    Closes-Bug: #1541471
    Upstream PR: https://github.com/rabbitmq/rabbitmq-server/pull/758
    (cherry picked from commit d64be61ada27a200894d7fd08c1803079cf3aed2)

[Alexey Galkin (agalkin)]

At now job https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.ha_neutron_destructive is failed. Waiting to apply #309928 fix.

[Ksenia Svechnikova (kdemina)]

Verify on ISO 9.0-250: https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_destructive/91/testReport/(root)/ha_neutron_destroy_controllers/

[Ksenia Svechnikova (kdemina)]

Verify on ISO fuel-10.0-153 https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ha_neutron_destructive/20/testReport/(root)/ha_neutron_destroy_controllers/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/312428

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/312439

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/313313

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Relewicz (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/312439
Reason: change with correct ID: https://review.openstack.org/313313

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/313315

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Relewicz (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/312428
Reason: change with correct ID: //review.openstack.org/313315

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313315
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3fecfb479b01e25d0c15ab31e4af06db2de8f30d
Submitter: Jenkins
Branch: stable/7.0

commit 3fecfb479b01e25d0c15ab31e4af06db2de8f30d
Author: Maciej Relewicz <email address hidden>
Date: Wed May 4 12:12:26 2016 +0200

    Stop process when rabbit is running but is not connected to master.

    It's should goes down due to avoid split brain.

    Change-Id: I4c51f8608702f2284d835ba9c3c9070b2c329ed8
    Closes-Bug: #1541471
    Upstream PR: https://github.com/rabbitmq/rabbitmq-server/pull/758

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/316053

[Dmitry Belyaninov (dbelyaninov)]

According to last run https://patching-ci.infra.mirantis.net/view/All/job/7.0.full.runner/32/console
described OSTF checks were successful.

But test was failed with another OSTF check:
https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.ha_neutron_destructive/33/testReport/%28root%29/ha_neutron_test_3_1_rabbit_failover/ha_neutron_test_3_1_rabbit_failover/

[Vitaly Sedelnik (vsedelnik)]

Removed from 8.0-mu-2 scope as the fix requires more work

[Denis Meltsaykin (dmeltsaykin)]

Won't fix for 6.x series as they are unsupported.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/8.0
Review: https://review.opendev.org/313313
Reason: This repo is retired now, no further work will get merged.