OCF Rabbit multiple failure tolerance mode does not clean up failures count after exiting monitor subroutine

[Vladimir Kuklin]

Fuel-Library SHA: 2c10f24398636e45f4661a4cacbeda70ec93f606

2015-11-04T17:15:39.108776+00:00 err: ERROR: p_rabbitmq-server: get_monitor(): 'rabbitmqctl list_channels' timed out 36 of max. 1 time(s) in a row and is not responding. The resource is failed.

[Dmitry Mescheryakov (dmitrymex)]

Guys, the OCF script cleans up failure count on RabbitMQ start here:
https://github.com/openstack/fuel-library/blob/master/files/fuel-ha-utils/ocf/rabbitmq#L1416-L1418

I.e. the script should have restarted the RabbitMQ long time ago (after the first failure) and that would clean up the counter. Can we see lrmd.log for the corresponding node? We need to understand what happened.

[Matthew Mosesohn (raytrac3r)]

This bug is missing a diagnostic snapshot and steps to reproduce

[Vladimir Kuklin (vkuklin)]

Okay, Matt, it is simple - inject return of error code into piece which does list_channels check and you will reproduce it easily.

Dmitry, the linest that you shared are not relevant.

Also, I meant that actually we should clear the counter after 1 successful command exit. Otherwise, although, you have your max_rabbitmqctl_timeout set to 3, it will fail infinitely after there were at least 3 1-time failures.

[Vladimir Kuklin (vkuklin)]

Or, Matt, you can try to introduce return of RAND % 2 and set max_rabbitmqctl_timeout to 3 and you will see that it will die forever after 3 monitor commands.

[Matthew Mosesohn (raytrac3r)]

Assigning back to Fuel Library to be retriaged

[Bogdan Dobrelya (bogdando)]

That is the operations impact? I see zero deployment impact, so this bug can be high only if this affects stability of operations.

[Dmitry Mescheryakov (dmitrymex)]

Vladimir, seems like I got your point. The script already drops counter to 0 if there was a successful rabbitmqctl run here:
  https://github.com/openstack/fuel-library/blob/7db9175477f35e961c0178814928e103fdd29d81/files/fuel-ha-utils/ocf/rabbitmq#L1127

Does that address your concern? I believe that the log line you posted in bug description is caused by a different bug: it is when OCF script returns error on 'monitor' call, but Pacemaker does not react and just continues to call 'monitor' again and again.

[Bogdan Dobrelya (bogdando)]

Reproduced in the bug 1528488 as well, see
2015-12-29T13:42:55.513788+00:00 node-17 lrmd err: ERROR: p_rabbitmq-server: get_monitor(): 'rabbitmqctl list_channels' timed out 23 of max. 1 time(s) in a row and is not responding. The resource is failed.

[Alexey Lebedeff (alebedev-a)]

Reproduced in the https://bugs.launchpad.net/mos/+bug/1533585

2016-01-13T07:25:38.524486+00:00 err: ERROR: p_rabbitmq-server: get_monitor(): 'rabbitmqctl list_channels' timed out 730 of max. 1 time(s) in a row and is not responding. The resource is failed.

[Dmitry Mescheryakov (dmitrymex)]

I've increased priority of the issue because it seems to cause the problem, when the script does not restart RabbitMQ while it is needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/266890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266890
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7379106d73e2d02a629b848bed1dce65acb9c7af
Submitter: Jenkins
Branch: master

commit 7379106d73e2d02a629b848bed1dce65acb9c7af
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 13 15:46:34 2016 +0300

    Reset master score if we decide to restart RabbitMQ on timeout

    Doing otherwise might not trigger the restart while it is clearly
    needed.

    Upstream PR: https://github.com/rabbitmq/rabbitmq-server/pull/560

    Change-Id: I480ebaddc98fa0784098efbf0c5ab8c512c8661d
    Closes-Bug: #1513421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/270778

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270778
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5f31ad4a21dd31a268120c866eca252b3683e38a
Submitter: Jenkins
Branch: stable/8.0

commit 5f31ad4a21dd31a268120c866eca252b3683e38a
Author: Dmitry Mescheryakov <email address hidden>
Date: Wed Jan 13 15:46:34 2016 +0300

    Reset master score if we decide to restart RabbitMQ on timeout

    Doing otherwise might not trigger the restart while it is clearly
    needed.

    Upstream PR: https://github.com/rabbitmq/rabbitmq-server/pull/560

    Change-Id: I480ebaddc98fa0784098efbf0c5ab8c512c8661d
    Closes-Bug: #1513421