failcounts never expire

[Andrea Ieri]

failure-timeout is currently unset in any monitored resource. This causes failcounts to accumulate indefinitely, even if the error condition that triggered them has long disappeared.
I think we should set a reasonable default expiration timeout, such as 1 minute.

[Ryan Beisner (1chb1n)]

Agree. I think failure-timeout as a new charm config option would make sense. We should have more conversation though about the default value.

https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_failure_response.html

[Andrea Ieri (aieri)]

Here is a patch for setting failure-timeout to 3 minutes by default: https://review.openstack.org/#/c/624720/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/624720
Committed: https://git.openstack.org/cgit/openstack/charm-hacluster/commit/?id=e28f8a9adc9f8ac1b3c1462d4f0724a4ddbc2257
Submitter: Zuul
Branch: master

commit e28f8a9adc9f8ac1b3c1462d4f0724a4ddbc2257
Author: Andrea Ieri <email address hidden>
Date: Wed Dec 12 15:32:59 2018 +0100

    Enable custom failure-timeout configuration

    As explained here[0], setting failure-timeout means that the cib will 'forget'
    that a resource agent action failed by setting failcount to 0:
    - if $failure-timeout seconds have elapsed from the last failure
    - if an event wakes up the policy engine (i.e. at the global resource
      recheck in an idle cluster)

    By default the failure-timeout will be set to 0, which disables the feature,
    however this change allows for tuning.

    [0] https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/#_failure_response

    Change-Id: Ia958a8c5472547c7cf0cb4ecd7e70cb226074b88
    Closes-Bug: #1802310