[library] Rabbitmq plugins do not work in HA mode

[Aleksandr Didenko]

{
    "api": "1.0",
    "astute_sha": "aa5aed61035a8dc4035ab1619a8bb540a7430a95",
    "auth_required": true,
    "build_id": "2014-07-28_10-30-48",
    "build_number": "367",
    "feature_groups": [
        "mirantis",
        "experimental"
    ],
    "fuellib_sha": "d1c7f67b3cf51978d3178c8666ea398f2477dcb5",
    "fuelmain_sha": "9adfbf5a52cedbdd16ec1a74f6c44c5b3419b87c",
    "nailgun_sha": "83cc9ed44ebc8dd97248483b6d414ebbc4cff3c0",
    "ostf_sha": "8c328521b1444f22c50463b9432193e20ed33813",
    "production": "docker",
    "release": "5.1"
}

Steps to reproduce:

1) Deploy any HA env (ubuntu or centos)
2) Run the following commands on any controller after deployment:
rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins list
crm resource restart p_rabbitmq-server

Expected result:
Plugins are loaded, rabbitmq listens 15672 and 55672 extra ports. "grep plugins /var/log/rabbitmq/ -r" shows something like this:
Server startup complete; 6 plugins started

Actual result:
Plugins are not loaded, rabbitmq listens only 41055 and 5673 ports. "grep plugins /var/log/rabbitmq/ -r" shows zero plugins:
Server startup complete; 0 plugins started.

[Aleksandr Didenko (adidenko)]

It looks like RABBITMQ_NODE_ONLY environment variable is the problem here.

RABBIT* environment vairables for the "beam" process started via init script (with working plugins):
RABBITMQ_PID_FILE=/var/run/rabbitmq/pid
RABBITMQ_NODE_PORT=5673

RABBIT* environment vairables for the "beam" process started via OCF (with not working plugins):
RABBITMQ_NODE_ONLY=1
RABBITMQ_NODENAME=rabbit@node-4
RABBITMQ_PID_FILE=/var/run/rabbitmq/pid
RABBITMQ_NODE_PORT=5673

For the experiment purpose I've remove RABBITMQ_NODE_ONLY=1 from OCF script and executed "crm resource restart p_rabbitmq-server" - rabbit has started with plugins:

/<email address hidden>: Server startup complete; 6 plugins started.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114282

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114282
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6274f28cd6b1fbb1a9af2be88450f0840ea533d3
Submitter: Jenkins
Branch: master

commit 6274f28cd6b1fbb1a9af2be88450f0840ea533d3
Author: Aleksandr Didenko <email address hidden>
Date: Thu Aug 14 18:58:39 2014 +0300

    Load rabbitmq plugins from OCF script

    Rabbitmq plugins do not work in HA mode since we start rabbitmq
    server with RABBITMQ_NODE_ONLY=1 which disables plugins.
    So we just load them with a piece of Erlang code sent to rabbitmq
    via "rabbitmqctl eval".

    Change-Id: Ib7e7d02bbf5f7185646c0fefbc4317d9bf5c4566
    Closes-bug: #1354026

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114526
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bce9efc948322f61e34ef69af6f775b83ead3a0f
Submitter: Jenkins
Branch: master

commit bce9efc948322f61e34ef69af6f775b83ead3a0f
Author: Aleksandr Didenko <email address hidden>
Date: Fri Aug 15 16:06:50 2014 +0300

    Fix plugins in HA mode

    Another problem with HA rabbitmq plugins. Since we run start_app
    and stop_app several times, plugins re-setup() causes problems.
    So we just need to check if plugins are already loaded before
    trying to setup() them again.

    Change-Id: I4e13a405b6895f92d3da3ac0fd69274951837e4e
    Closes-bug: #1354026