keystone-fernet cron job not triggering, results in keys not rotating

[joel waddell]

**Bug Report**

What happened:

in the keystone-fernet container you can see the cronjob is setup but does not seem to be working. I can manually run /usr/bin/fernet-rotate.sh and that works.

```
(keystone-fernet)[root@ddc-control-1 fernet-keys]# cat /var/spool/cron/crontabs/root/fernet-cron
PATH=/var/lib/kolla/venv/bin:/usr/bin:/bin

0 0 * * 0 /usr/bin/fernet-rotate.sh

(keystone-fernet)[root@ddc-control-1 fernet-keys]# date
Tue 06 Oct 2020 05:53:15 PM UTC

```

You can see the keys below were not rotated:

```
(keystone-fernet)[root@ddc-control-1 fernet-keys]# ls -ahl
total 20K
drwxrwx--- 2 keystone keystone 4.0K Sep 28 17:27 .
drwxr-xr-x 1 keystone keystone 4.0K Sep 28 17:43 ..
-rw------- 1 keystone keystone 44 Sep 28 17:27 0
-rw------- 1 keystone keystone 44 Sep 25 12:32 3
-rw------- 1 keystone keystone 44 Sep 25 16:07 4

```

in my /etc/kolla/globals.yml I have set for troubleshooting purposes. But I had it at default for the two values prior to this change.

```
fernet_token_expiry: 172800
fernet_token_allow_expired_window: 345600

```

This causes a problem when I try to restart the keystone containers because the get caught in a restart loop because the primary fernet key is expired. similar to what is reported in
https://bugs.launchpad.net/kolla-ansible/+bug/1895723

What you expected to happen:

    I expected the fernet-rotate.sh script to run on sunday at 00:00.

How to reproduce it:

    considering the issue is the script is not running at the specified date and time, i'm not sure how to reproduce it because I don't know the cause. However I'm open to suggestions.

**Environment**:
* OS (e.g. from /etc/os-release):
  Ubuntu server 20.04.1 LTS

* Kernel (e.g. `uname -a`):
  5.4.0-48-generic

* Docker version if applicable (e.g. `docker version`):
  Docker version 19.03.13, build 4484c46d9d

* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release):
  git head 7e91cb8f542493a35ec8853f6b65e3e8f08c1067

* Docker image Install type (source/binary):
  source

* Docker image distribution:
  ubuntu
* Are you using official images from Docker Hub or self built?
  docker hub

* If self built - Kolla version and environment used to build:

* Share your inventory file, globals.yml and other configuration files if relevant
  uploaded.

[Michal Nasiadka (mnasiadka)]

Hello Joel, can you please try to apply that patch on your kolla-ansible https://review.opendev.org/#/c/756083/ - it will add some logging to the script, and we could investigate if it's really running or not.

[joel waddell (maasmaaster)]

Yeah I can do that. What is the best way to apply the patch? Is it just to git pull > pip install ./kolla-ansible > then kolla-ansible deploy?

[Mark Goddard (mgoddard)]

Joel, find the Download button in the top right of the review page. It has several options for getting the patch. You might want to try a cherry-pick if you are not using the master branch.

[joel waddell (maasmaaster)]

Ok, I have applied the patch, the cronjobs are set for wednesday and sunday at 00:00:00 so it looks like the next cron is on sunday. I'll post the contents of that log file on monday when i'm in the office. Unless you guys think it's a good idea to try and edit one of the containers /var/spool/cron/crontabs/root/fernet-cron for a earlier date, not sure the best way to do that so I would need a bit of help if you guys wanted to go that route.

[joel waddell (maasmaaster)]

It looks like the cron job is not firing off. I checked to see if /var/log/kolla/keystone/fernet-rotate.log exists and the file is missing. see output below showing that.

(keystone-fernet)[root@ddc-control-1 fernet-keys]# cat /var/log/kolla/keystone/
apache-access.log keystone-apache-public-access.log keystone-apache-public-access.log.4.gz keystone.log
apache-error.log keystone-apache-public-access.log.1 keystone-apache-public-access.log.5.gz
keystone-apache-admin-access.log keystone-apache-public-access.log.2.gz keystone-apache-public-access.log.6.gz
keystone-apache-admin-error.log keystone-apache-public-access.log.3.gz keystone-apache-public-error.log

(keystone-fernet)[root@ddc-control-1 fernet-keys]# cat /var/spool/cron/crontabs/root/fernet-cron

PATH=/var/lib/kolla/venv/bin:/usr/bin:/bin

0 0 * * 0 /usr/bin/fernet-rotate.sh >> /var/log/kolla/keystone/fernet-rotate.log 2>&1
(keystone-fernet)[root@ddc-control-1 fernet-keys]#

(keystone-fernet)[root@ddc-control-1 fernet-keys]# date
Mon 19 Oct 2020 01:03:36 PM UTC

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757076
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=07d42bff01c811daf6e19b4c4984950243613157
Submitter: Zuul
Branch: master

commit 07d42bff01c811daf6e19b4c4984950243613157
Author: Michal Nasiadka <email address hidden>
Date: Fri Oct 9 12:25:40 2020 +0200

    Fix fernet cron path on Ubuntu/Debian

    The correct path according to Ubuntu cron manpage [1] is
    /var/spool/cron/crontabs/$USER

    [1]: http://manpages.ubuntu.com/manpages/trusty/man8/cron.8.html

    Closes-Bug: #1898765
    Change-Id: Id5fc354e3e32cae2468cd2557a2967859e3b4e16

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/759217

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/759218

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/759219

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759219
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f97a0eea124ad32b8a53039d44b58e1271c5afd4
Submitter: Zuul
Branch: stable/stein

commit f97a0eea124ad32b8a53039d44b58e1271c5afd4
Author: Michal Nasiadka <email address hidden>
Date: Fri Oct 9 12:25:40 2020 +0200

    Fix fernet cron path on Ubuntu/Debian

    The correct path according to Ubuntu cron manpage [1] is
    /var/spool/cron/crontabs/$USER

    [1]: http://manpages.ubuntu.com/manpages/trusty/man8/cron.8.html

    Closes-Bug: #1898765
    Change-Id: Id5fc354e3e32cae2468cd2557a2967859e3b4e16
    (cherry picked from commit 07d42bff01c811daf6e19b4c4984950243613157)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759218
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=dc497edc7c71047f6d803ae8d6a13d16fe9c4e1e
Submitter: Zuul
Branch: stable/train

commit dc497edc7c71047f6d803ae8d6a13d16fe9c4e1e
Author: Michal Nasiadka <email address hidden>
Date: Fri Oct 9 12:25:40 2020 +0200

    Fix fernet cron path on Ubuntu/Debian

    The correct path according to Ubuntu cron manpage [1] is
    /var/spool/cron/crontabs/$USER

    [1]: http://manpages.ubuntu.com/manpages/trusty/man8/cron.8.html

    Closes-Bug: #1898765
    Change-Id: Id5fc354e3e32cae2468cd2557a2967859e3b4e16
    (cherry picked from commit 07d42bff01c811daf6e19b4c4984950243613157)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759217
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5d2c8d5a1d974dd437a6cbce1aa790a14f768d45
Submitter: Zuul
Branch: stable/ussuri

commit 5d2c8d5a1d974dd437a6cbce1aa790a14f768d45
Author: Michal Nasiadka <email address hidden>
Date: Fri Oct 9 12:25:40 2020 +0200

    Fix fernet cron path on Ubuntu/Debian

    The correct path according to Ubuntu cron manpage [1] is
    /var/spool/cron/crontabs/$USER

    [1]: http://manpages.ubuntu.com/manpages/trusty/man8/cron.8.html

    Closes-Bug: #1898765
    Change-Id: Id5fc354e3e32cae2468cd2557a2967859e3b4e16
    (cherry picked from commit 07d42bff01c811daf6e19b4c4984950243613157)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.