RabbitMQ OCF node uptime is not always reset in the CIB

[Bogdan Dobrelya]

Source bug https://bugs.launchpad.net/fuel/+bug/1529875

What I found is that the uptime value was not reset for the node-3 after it was rebooted, which is wrong and impacts election of new masters very badly:

node-4 lrmd:
before node-3 reset
2015-12-29T13:34:44.242420+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (4490) with node-9.domain.tld (4488)
2015-12-29T13:34:44.317439+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (4490) with node-3.domain.tld (7142)
after node-3 reset
2015-12-29T13:38:23.253014+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (4709) with node-9.domain.tld (4707)
2015-12-29T13:38:23.359804+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (4709) with node-3.domain.tld (7361)

The crm_mon -fotAW -1 also shows the value for the node's rabbit-start-time is not being reset after the non-graceful shutdown.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/262572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262572
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d833b3ac710a950affedd7cd32c0abaef81944e0
Submitter: Jenkins
Branch: master

commit d833b3ac710a950affedd7cd32c0abaef81944e0
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Dec 30 18:08:46 2015 +0100

    Ensure rabbit node uptime is reset in the CIB for OCF resource

    * Add ocf_run wrappers and info log messages for CIB attribute events
    * Move "fast" CIB attribute updates before "heavy" operations like
      start/stop/wait to ensure CIB consistent even if the timeouts
      exceeded for the ops
    * Delete master and start time attributes from CIB on action_start
      to ensure the correct rabbit nodes uptime evaluation for new
      master elections for corresponding pacemaker resources
    * For post-demote notify and action_demote() delete the master
      attribute from CIB as well.
    * For post-start notify, update the start time in the CIB even when
      the node is already clustered. Otherwise it would remain running
      in cluster w/o the start time registered, which affects the new
      master elections badly.

    Upstream RR https://github.com/rabbitmq/rabbitmq-server/pull/524
    Closes-bug: #1530150

    Change-Id: I9db3c819031cef620377b4fee08ea92e90b11c70
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/267507

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267507
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=51b4072521684a19c2ba7d394cf8d42b191ced93
Submitter: Jenkins
Branch: stable/8.0

commit 51b4072521684a19c2ba7d394cf8d42b191ced93
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Dec 30 18:08:46 2015 +0100

    Ensure rabbit node uptime is reset in the CIB for OCF resource

    * Add ocf_run wrappers and info log messages for CIB attribute events
    * Move "fast" CIB attribute updates before "heavy" operations like
      start/stop/wait to ensure CIB consistent even if the timeouts
      exceeded for the ops
    * Delete master and start time attributes from CIB on action_start
      to ensure the correct rabbit nodes uptime evaluation for new
      master elections for corresponding pacemaker resources
    * For post-demote notify and action_demote() delete the master
      attribute from CIB as well.
    * For post-start notify, update the start time in the CIB even when
      the node is already clustered. Otherwise it would remain running
      in cluster w/o the start time registered, which affects the new
      master elections badly.

    Upstream RR https://github.com/rabbitmq/rabbitmq-server/pull/524
    Closes-bug: #1530150

    Change-Id: I9db3c819031cef620377b4fee08ea92e90b11c70
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit d833b3ac710a950affedd7cd32c0abaef81944e0)

[Andrey Sledzinskiy (asledzinskiy)]

verified on 8.0-506

[Vladimir (vushakov)]

Verified on:
    Fuel 9.0 ISO #142