token flush cronjob not being enabled on leader unit

[Andrea Ieri]

I have seen in a few xenial-queens clouds that the token flush cronjob is completely disabled:

jujumanage@MAAS:~$ juju run --application keystone -- 'is-leader; cat /etc/cron.d/keystone-token-flush'
- Stdout: |
    False
    # Purge expired tokens from the keystone database hourly, per OpenStack installation guide.
    # Current unit is not the leader unit. Token flush will be managed by the leader unit.
  UnitId: keystone/0
- Stdout: |
    False
    # Purge expired tokens from the keystone database hourly, per OpenStack installation guide.
    # Current unit is not the leader unit. Token flush will be managed by the leader unit.
  UnitId: keystone/1
- Stdout: |
    True
    # Purge expired tokens from the keystone database hourly, per OpenStack installation guide.
    # Current unit is not the leader unit. Token flush will be managed by the leader unit.
  UnitId: keystone/2

The above happens with charm versions 289, 290, and 291.

The cronjob does however work fine in a xenial-ocata cloud using the 18.05 series, so the bug has probably been introduced somewhere between 18.05 and 18.11

[Andrea Ieri (aieri)]

It happens on xenial-pike with keystone v285 as well.

[Andrea Ieri (aieri)]

Subscribed field-medium, as this affects a lot of clouds we manage and we had observed in the past pretty substantial slowdowns when the flush was not enabled

[Andrea Ieri (aieri)]

Since the question will probably come up: token-provider is not set in the clouds affected by this bug.
If I understand correctly, that should yield uuid tokens, causing token_flush to be set to True in the context (for the leader unit).

[James Page (james-page)]

The check in the context looks to see if fernet is enabled and if the unit is the elected leader (using the old style is_elected_leader function).

I think this is wrong - it should be relying on is_leader and juju for this

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/638431

[James Page (james-page)]

Confirmed - the fernet token support has a broken check.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638431
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=1a07a7e65731ee2585818f998638ad71d91662a0
Submitter: Zuul
Branch: master

commit 1a07a7e65731ee2585818f998638ad71d91662a0
Author: James Page <email address hidden>
Date: Thu Feb 21 14:58:58 2019 +0000

    Fix issue with crontab enablement

    The token flush and token rotate crontabs are re-written when the
    leader unit changes inline with Juju leadership management.

    Align contexts used to generate crontabs with Juju leadership
    status, rather than corosync/pacemaker.

    Correct use of OpenStackCompareReleases to ensure that releases
    between ocata and queens don't automatically enable fernet
    token behaviour.

    Change-Id: I6db8d006ceac7b61e69f547682c5a49d876cfec6
    Closes-Bug: 1816807

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/638609

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638609
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=f8c4a81803454b7a4dc8c6b6881ffac925f3e9cb
Submitter: Zuul
Branch: stable/18.11

commit f8c4a81803454b7a4dc8c6b6881ffac925f3e9cb
Author: James Page <email address hidden>
Date: Thu Feb 21 14:58:58 2019 +0000

    Fix issue with crontab enablement

    The token flush and token rotate crontabs are re-written when the
    leader unit changes inline with Juju leadership management.

    Align contexts used to generate crontabs with Juju leadership
    status, rather than corosync/pacemaker.

    Correct use of OpenStackCompareReleases to ensure that releases
    between ocata and queens don't automatically enable fernet
    token behaviour.

    Change-Id: I6db8d006ceac7b61e69f547682c5a49d876cfec6
    Closes-Bug: 1816807
    (cherry picked from commit 1a07a7e65731ee2585818f998638ad71d91662a0)