Make the rabbitmq OCF to check ha sync for graceful stop, demote actions

[Bogdan Dobrelya]

In case of a graceful action demote of a master or a slave stopping, it could be nice to ensure the queues owned by this node are fully synchronized elsewhere. Otherwise, there could be data loss.

The graceful cases are when the operator stops or demotes the resource
by request.

[Matthew Mosesohn (raytrac3r)]

From Bogdan:
This task may be done by analyzing rabbitmqctl list queues slave_pids syncronized_slave_pids and waiting for sync until queues are all synced. The loop or method should be called wait_for_all_queues_in_sync. The loop should run until 50% of timeout interval for stop or demote action, defined by OCF_RESKEY_CRM_meta_timeout.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233019

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233019
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9ca5b6709cd18c8134cd8feaccf937ef49a625b4
Submitter: Jenkins
Branch: master

commit 9ca5b6709cd18c8134cd8feaccf937ef49a625b4
Author: Matthew Mosesohn <email address hidden>
Date: Mon Oct 5 15:48:33 2015 +0300

    Wait for rabbitmq sync before stop/demote actions

    Added new OCF key stop_time (corresponding to start_time)
    Added wait_sync function which tries until start_time/2
    for queues on stopped/demoted node to reach synced state.

    Added optional [-t timeout] to su_rabbit_cmd function to
    provide arbitrary timeout

    Change-Id: Iae2211b3d477a9603a58d5eacb12e0fba924861a
    Closes-Bug: #1464637

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/235422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235422
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0204868a14ac0aa310fc135135a88b6aec576d8c
Submitter: Jenkins
Branch: master

commit 0204868a14ac0aa310fc135135a88b6aec576d8c
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Oct 15 17:15:30 2015 +0200

    Fix the timeout arg for the su_rabbit_cmd

    And fix local bashisms as a little bonus
    Upstream patch https://github.com/rabbitmq/rabbitmq-server/pull/374

    Related-bug: #1464637

    Change-Id: I13189de9f8abce23673c031d11132e495e1972e3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[ElenaRossokhina (esolomina)]

Steps for verification/reproducing are unclear
@bogdando, could you please update description with additional information how fix can be checked?

[Bogdan Dobrelya (bogdando)]

Please do not move to incomplete bugs with commits submitted. Such bags may become invalid by timeouts, while fixed in fact

[Bogdan Dobrelya (bogdando)]

The test case:
1) For a running rabbit resource master node-X, issue demote and stop actions as:

a) "OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/rabbitmq/rabbitmq-server-ha demote"

- It is expected to get log records to the console like these below, and in the following order:

demote: action begin
demote: waiting.*to sync
demote: Execute stop_app with timeout
demote: action end

b) next run
"OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/rabbitmq/rabbitmq-server-ha stop"

- It is expected to get log records to the console like these below, and in the following order:

stop: action begin
stop: waiting.*to sync
stop_server_process(): Execute stop with timeout
stop: action end

[Tatyanka (tatyana-leontovich)]

570 verified