Rabbit OCF - master status caching error

[Alexey Lebedeff]

Single monitor run reports several nodes as a master. Like this:

2016-01-29T01:50:17.168543+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. master is node-2.test.domain.local
2016-01-29T01:50:17.841360+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. master is node-1.test.domain.local

dmitriyme pointed out that it happens nearby of https://github.com/openstack/fuel-library/blob/bde7c0d4a64c02350ef169060d0218fbf46569c5/files/fuel-ha-utils/ocf/rabbitmq#L1394 - probably status_master should be initialized at every loop iteration there

[Bogdan Dobrelya (bogdando)]

Please provide logs

[Bogdan Dobrelya (bogdando)]

You're right, no logs needed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275312

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275312
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6c0364ab335445d27ab557f3686ee32442607c6f
Submitter: Jenkins
Branch: master

commit 6c0364ab335445d27ab557f3686ee32442607c6f
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 2 17:31:34 2016 +0100

    Fix uninitialized status_master

    Fix multiple nodes may be reported in logs as the running master

    Closes-bug: #1540936

    Change-Id: Ic2dfe7b2ba657b9bf06d97f49ddb4b69f2f4e063
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Andrey Maximov (maximov)]

mos team supports rabbitmq ha scripts.

[Andrey Maximov (maximov)]

so I set area-mos tag

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275772
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b0f86ec4e8580b17af550defc335e9f6ba8bd5f5
Submitter: Jenkins
Branch: stable/8.0

commit b0f86ec4e8580b17af550defc335e9f6ba8bd5f5
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 2 17:31:34 2016 +0100

    Fix uninitialized status_master

    Fix multiple nodes may be reported in logs as the running master

    Closes-bug: #1540936

    Change-Id: Ic2dfe7b2ba657b9bf06d97f49ddb4b69f2f4e063
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 6c0364ab335445d27ab557f3686ee32442607c6f)

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 539. The issue is fixed.

ISO details:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "539"
  build_id: "539"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "87dfb6bc25d4650264f09c338ed77c21a3d6fe87"

[Sofiia Andriichenko (sandriichenko)]

[root@nailgun ~]# cd /var/log/remote
[root@nailgun remote]# find . |grep lrmd
./node-39.test.domain.local/lrmd.log
./node-40.test.domain.local/lrmd.log
./node-38.test.domain.local/lrmd.log
[root@nailgun remote]# grep -i 'master' ./node-38.test.domain.local/lrmd.log
2016-03-22T09:25:07.346625+00:00 info: INFO: p_rabbitmq-server[30860]: get_monitor(): also checking if we are master.
2016-03-22T09:25:08.805039+00:00 info: INFO: p_rabbitmq-server[30860]: get_monitor(): master attribute is 0
...

[root@nailgun remote]# grep -i 'master is' ./node-39.test.domain.local/lrmd.log
2016-03-22T09:12:44.655445+00:00 info: INFO: p_rabbitmq-server[32475]: get_monitor(): master is node-38.test.domain.local
....
[root@nailgun remote]# grep -i 'master is' ./node-40.test.domain.local/lrmd.log
2016-03-22T10:21:50.148183+00:00 info: INFO: p_rabbitmq-server[5888]: get_monitor(): master is node-38.test.domain.local
...

[Sofiia Andriichenko (sandriichenko)]

http://srv41-bud.infra.mirantis.net/fuelweb-iso/fuel-9.0-68-2016-03-16_08-00-00.iso.torrent