Key rotation performed regardless of lead unit being paused or not

[Frode Nordahl]

As can be seen in the artifacts from a test failure here:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-keystone/593266/1/1825/index.html

Traceback (most recent call last):
  File "/var/lib/jenkins/checkout/0/keystone/.tox/func/lib/python3.5/site-packages/zaza/charm_tests/keystone/tests.py", line 130, in test_key_distribution_and_rotation
    pprint.pformat(unit_repo)))
zaza.utilities.exceptions.KeystoneKeyRepositoryError: expect: "OrderedDict([('/etc/keystone/credential-keys/',
              OrderedDict([('0',
                            'uNMjfqs7nPdMHeAYCeUoNcuH-jcd5KlL9eltGk-QiUI='),
                           ('1',
                            'JaUJG9mXt__ahzNLtinhquoqygIDZwWo-BSaSJczih0=')])),
             ('/etc/keystone/fernet-keys/',
              OrderedDict([('0',
                            'HmfrbKuHm_PTrBKvlEhGHPrao9BG5giYpmAOrfQXSpY='),
                           ('10',
                            'uEqakkPg-zd-ZLmw0KotdsnJ2oqwhJlGJ5GU6Scisgw='),
                           ('11',
                            'gVUe-laRj7GP9fltRgjIDVMJeua7N_wwUFDJrguwtMI=')]))])" actual(keystone/2): "OrderedDict([('/etc/keystone/credential-keys/',
              OrderedDict([('0',
                            'uNMjfqs7nPdMHeAYCeUoNcuH-jcd5KlL9eltGk-QiUI='),
                           ('1',
                            'JaUJG9mXt__ahzNLtinhquoqygIDZwWo-BSaSJczih0=')])),
             ('/etc/keystone/fernet-keys/',
              OrderedDict([('0',
                            'HmfrbKuHm_PTrBKvlEhGHPrao9BG5giYpmAOrfQXSpY='),
                           ('10', ''),
                           ('11',
                            'gVUe-laRj7GP9fltRgjIDVMJeua7N_wwUFDJrguwtMI=')]))])"

If you examine the juju crashdump you will see that the keys are fully rotated and written to disk at a later time, so our test has caught it in the middle of a rotation.

There are two possible solutions to this intermittent test failure:
1. Pause the lead unit before checking that rotation has taken place.
2. Only list key file names and take existence of keys with higher index than what is created on initial setup as proof of successful rotation.

While investigating these options I came across the fact that the charm currently will rotate keys regardless of it being paused, and that in itself is a bug, thus the filing of this bug report.

[Frode Nordahl (fnordahl)]

Review: https://review.openstack.org/#/c/593532/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/593532
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=93db01848530f8e160509ecc78ea9520626df21f
Submitter: Zuul
Branch: master

commit 93db01848530f8e160509ecc78ea9520626df21f
Author: Frode Nordahl <email address hidden>
Date: Mon Aug 20 09:37:09 2018 +0200

    Do not rotate keys when lead unit is paused

    Closes-Bug: #1787719
    Change-Id: I0557803e90d8ec52271f01e5e7276d2db8338ce2