RedisDriver _stop() method doesn't release locks

[Bob Haddleton]

The RedisDriver _stop() method doesn't release locks because it pop()s them out of the _acquired_locks set before calling lock.release(), and the RedisLock release() method returns False without releasing the lock if it doesn't find the lock in the coordinator's _acquired_locks set.

    def _stop(self):
        while self._acquired_locks:
            lock = self._acquired_locks.pop()
            try:
                lock.release()

RedisLock():
    def release(self):
        with self._exclusive_access:
            if not self.acquired:
                return False
            with _translate_failures():
                try:
                    self._lock.release()

Since the lock has been popped out of the set, it is never released and eventually times out.

This delays the election of a new leader for the group until the lock times out.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/636038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/636038
Committed: https://git.openstack.org/cgit/openstack/tooz/commit/?id=842c7404b2d01c5dd718c14bb55f0cd47471f0e1
Submitter: Zuul
Branch: master

commit 842c7404b2d01c5dd718c14bb55f0cd47471f0e1
Author: Bob.Haddleton <email address hidden>
Date: Sat Feb 9 21:24:54 2019 -0600

    RedisLock release() should not check if the lock has been acquired

    The RedisDriver coordinator _stop() method removes the lock from the
    _acquired_locks set before calling the release() method on the lock,
    so the release() method should not check to see if the lock is in the
    set, since it never will be.

    This was preventing the coordinator's _stop() method from releasing
    the locks, so they had to timeout before another group member could
    take over as leader.

    Change-Id: Ie23d88f1a05983d7465f18a329cabc23d789a628
    Closes-Bug: 1815330

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tooz 1.67.2 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tooz/+/848653

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tooz/+/848653
Committed: https://opendev.org/openstack/tooz/commit/85d0a8b2e09e37639ad29bb3bbcbbad55719dd8e
Submitter: "Zuul (22348)"
Branch: stable/train

commit 85d0a8b2e09e37639ad29bb3bbcbbad55719dd8e
Author: Bob.Haddleton <email address hidden>
Date: Sat Feb 9 21:24:54 2019 -0600

    RedisLock release() should not check if the lock has been acquired

    The RedisDriver coordinator _stop() method removes the lock from the
    _acquired_locks set before calling the release() method on the lock,
    so the release() method should not check to see if the lock is in the
    set, since it never will be.

    This was preventing the coordinator's _stop() method from releasing
    the locks, so they had to timeout before another group member could
    take over as leader.

    Change-Id: Ie23d88f1a05983d7465f18a329cabc23d789a628
    Closes-Bug: 1815330
    (cherry picked from commit 842c7404b2d01c5dd718c14bb55f0cd47471f0e1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tooz train-eol release.