fuel-rabbit-fence is not able to manage its pid file

[Dmitry Burmistrov]

Detailed bug description:
 9.0 ISO
 Syslog on nodes with rabbit-fence is full of this errors: http://paste.openstack.org/show/561296/
 In case of incorrect pid-file rabbit-fence can't work - it is being respawned by upstart (you can check "service fuel-rabbit-fence status" under "watch" tool).
 It affects re-applying "rabbitmq/rabbitmq.pp" modular (when services are refreshed) - command "rabbitmqctl -q list_users" starts failing sometimes.
Steps to reproduce:
 Deploy cluster.
 echo "$RANDOM" > /var/run/rabbitmq/rabbit-fence.pid
 service fuel-rabbit-fence restart
 watch service fuel-rabbit-fence status
Expected results:
 watch is showing the same PID (process is not falling down)
Actual result:
 process is respawning
Workaround:
 add "rm -f /var/run/rabbitmq/rabbit-fence.pid" at the end of 'pre-start script' in /etc/init/fuel-rabbit-fence.conf
Description of the environment:
 Operation system: Ubuntu 14.04

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/357893

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/357895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/357895
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=41b1294f9a496e5031b34abcd7f4ad9bd84c027b
Submitter: Jenkins
Branch: stable/mitaka

commit 41b1294f9a496e5031b34abcd7f4ad9bd84c027b
Author: dmburmistrov <email address hidden>
Date: Fri Aug 19 16:36:26 2016 +0300

    Refactor lock-file handling

    Currently fuel-rabbit-fence is not able
    to manage its pid-file properly.
    Current implementation was refactored.
    "fcntl" is used.

    Closes-bug: #1614963

    Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/357893
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5501d2d14cd0b00f6669b017497c7e0f31780869
Submitter: Jenkins
Branch: master

commit 5501d2d14cd0b00f6669b017497c7e0f31780869
Author: dmburmistrov <email address hidden>
Date: Fri Aug 19 16:36:26 2016 +0300

    Refactor lock-file handling

    Currently fuel-rabbit-fence is not able
    to manage its pid-file properly.
    Current implementation was refactored.
    "fcntl" is used.

    Closes-bug: #1614963

    Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

[Sofiia Andriichenko (sandriichenko)]

Verified on MOS 9.1 (snapshot #209)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.

[Rene Soto (rsoto)]

Customer also just experienced this exact issue in a MOS 8.0 environment. Identical error messages were flooding /var/log/syslog on a specific controller node. The same fix seems to have resolved the issue in this particular environment as well.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/8.0
Change author: Dmitry Burmistrov <email address hidden>
Review: https://review.fuel-infra.org/37194

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Dmitry Burmistrov <email address hidden>
Review: https://review.fuel-infra.org/37436

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/37436
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: b083396199e62017f7197b7011fcb13c721a0e9a
Author: Dmitry Burmistrov <email address hidden>
Date: Tue Dec 12 12:54:28 2017

Refactor lock-file handling

Currently fuel-rabbit-fence is not able
to manage its pid-file properly.
Current implementation was refactored.
"fcntl" is used.

Closes-bug: #1614963

Change-Id: Ia5f58c3eb964d25c733cc91a6a5373ddbf193e77

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis V. Meltsaykin <email address hidden> on branch: stable/8.0
Review: https://review.fuel-infra.org/37194
Reason: I must have recommended the wrong branch, sorry.

[Vladimir Jigulin (vjigulin)]

Cannot reproduce in different labs. Closing on the basis of Rene Soto (rsoto) message #9.