RabbitMQ OCF scripts kill all beam processes hitting the Murano as well

[Serg Melikyan]

We have issue with corosync/pacemaker scripts that detects instance of second RabbitMQ (used by Murano for interaction with VM-side deployed agent) as instance of OpenStack RabbitMQ (used by all OpenStack components) and no proper failover happens for OpenStack RabbitMQ configured in HA.

This issue was found on MOS 5.1

[Tatyanka (tatyana-leontovich)]

Serge could you please clarify steps to reproduce

[Bogdan Dobrelya (bogdando)]

We use killall -9 beam in OCF script for rabbit which is wrong approach and affects all rabbit's instance including the one for Murano

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141359

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/141361

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/141362

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141359
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cda822d7f3690f4439ecd834b6af0cede3fc23fb
Submitter: Jenkins
Branch: master

commit cda822d7f3690f4439ecd834b6af0cede3fc23fb
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Dec 12 13:58:15 2014 +0100

    Fix killing of Rabbit beam process for OCF scripts

    W/o this patch, 'killall beam' commands kill all instances
    of rabbitmq including the one for Murano.

    The solution is to use kill_rmq_and_remove_pid()
    procedure to kill it by pidfile instead.

    Closes-bug: #1400670

    Change-Id: Ic8bb0b4ea22df22784bdb8fca0ccd0f2cb02bc9e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141361
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=87ad22e7965b770bd22c8a576997828afdbb0a38
Submitter: Jenkins
Branch: stable/6.0

commit 87ad22e7965b770bd22c8a576997828afdbb0a38
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Dec 12 13:58:15 2014 +0100

    Fix killing of Rabbit beam process for OCF scripts

    W/o this patch, 'killall beam' commands kill all instances
    of rabbitmq including the one for Murano.

    The solution is to use kill_rmq_and_remove_pid()
    procedure to kill it by pidfile instead.

    Closes-bug: #1400670

    Change-Id: Ic8bb0b4ea22df22784bdb8fca0ccd0f2cb02bc9e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141362
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=01da95166bf7d5322fa206346f0ef6e05c7f4338
Submitter: Jenkins
Branch: stable/5.1

commit 01da95166bf7d5322fa206346f0ef6e05c7f4338
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Dec 12 13:58:15 2014 +0100

    Fix killing of Rabbit beam process for OCF scripts

    W/o this patch, 'killall beam' commands kill all instances
    of rabbitmq including the one for Murano.

    The solution is to use kill_rmq_and_remove_pid()
    procedure to kill it by pidfile instead.

    Closes-bug: #1400670

    Change-Id: Ic8bb0b4ea22df22784bdb8fca0ccd0f2cb02bc9e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Timur Nurlygayanov (tnurlygayanov)]

We will verify this issue on MOS 6.0 rc3.

[Timur Nurlygayanov (tnurlygayanov)]

Verified on ISO MOS 6.0 #56 rc3:
{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Steps To Verify:
1. Deploy CentOS HA with 1 controller, install Murano.
2. Add 2 new controllers and redeploy the environment.
3. Login to controller node via ssh.
4. Execute command: 'ps ax | grep rabbit' and verify that we have several RabbitMQ services.
5. Stop main RabbitMQ service: 'service rabbitmq-server stop'
6. Verify that main rabbit MQ service stopped and only murano RabbitMQ service are still alive: 'ps ax | grep rabbit'