Cron job for key rotation not running in keystone_fernet container for source images

[Eric Miller]

Kolla Ansible 8.0.1 (Stein, built from source) deployment on CentOS
Kernel - 4.18.16-1.el7.elrepo.x86_64

I noticed that the dates on the fernet keys were somewhat old, and it appears that the cron job isn't running or something is failing, but without an error anywhere.

I can connect to the container (docker exec -it) and run the script (/usr/bin/fernet-rotate.sh), and it runs fine.

crond is running (as root), and the schedule is correct (I even forced the schedule of one of the controllers to a couple minutes from now, and it still didn't execute the script).

No errors appear in the docker logs for this container (the keystone_fernet container).

Maybe there is a permissions issue elsewhere (at the container level perhaps)? Anyone else have this issue?

Any ideas?

Eric

[Mark Goddard (mgoddard)]

I haven't noticed this issue, although we did recently make changes to keystone key rotation. Did you get to the bottom of it?

[Eric Miller (erickmiller)]

I saw the changes, but I don't think these have anything to do with the cron job not running.

(I just tried the items below again to be sure I wasn't going crazy)

I can connect to the container and run the script using:
ssh -qt controller001 docker exec -u root -it keystone_fernet /usr/bin/fernet-rotate.sh

and it runs fine.

However, I can run "crontab -e" when logged-in to the container using:
ssh -t controller001 docker exec -u root -it keystone_fernet bash

and view the cron job as (note that I forced the hour/minute to a time that was going to happen in a few minutes during this test):
25 15 * * * /usr/bin/fernet-rotate.sh

and the script did NOT run. Looking at the processes in the container indicates crond is running:

(keystone-fernet)[root@controller001 /]# ps auxgww | grep cron
root 6 0.0 0.0 22740 2684 ? S Oct14 0:01 crond -s -n

So I can only imagine that it is related to permissions somewhere (cron not being able to run the script?).

There is no /var/log/cron in the keystone_fernet container, like there is on non-container machines, so I don't know if there is anything being logged anywhere.

Docker logs just show that crond is run at container start:

+ echo 'Running command: '\''crond -s -n'\'''
+ exec crond -s -n

Eric

[Mark Goddard (mgoddard)]

The keystone_fernet container runs as the root user (this isn't ideal, there's a bug about it somewhere), so it shouldn't be permissions.

What about timezones? We do mount /etc/localtime into the container, but it's worth checking.

[Eric Miller (erickmiller)]

I should have mentioned that the cron task "never" runs, regardless of whether it is set to repeat at regular intervals - so timezone wouldn't matter. The keystone keys hadn't rotated for weeks (probably when we upgraded to Stein).

Now, what is interesting... if I create a cron task of:
* * * * * touch /tmp/crontest

The file "crontest" _is_ created! So, cron must be running. Just not the key rotation script.

So, I looked at some other systems, and we have this at the top of the crontab file:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I added this to the keystone_fernet container, to see if it made a difference. It did not.

I set both the cron jobs to the same time:
16 0 * * * /usr/bin/fernet-rotate.sh
16 0 * * * touch /tmp/crontest

and the crontest file was updated, but keys were not rotated. This also indicates that I have the timezone set correctly.

[Mark Goddard (mgoddard)]

I have a system running centos/source/master containers, and confirmed tokens are not being rotated. I modified the crontab file to include the following (set to trigger at 11:07AM):

07 11 * * * /usr/bin/bash -x -c "/usr/bin/fernet-rotate.sh >/tmp/rotate.log 2>&1"

I then checked /tmp/rotate.log, and found this:

/usr/bin/fernet-rotate.sh: line 3: keystone-manage: command not found

keystone-manage is installed in the kolla virtualenv for source images, which is presumably not in the PATH used by cron:

/var/lib/kolla/venv/bin/keystone-manage

There is a tangential issue here which is that the lack of logs made this hard to debug. We run crond with a -s argument, which should log to syslog.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692990
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d09ee49afc19ad5e752755e14ac46b9af1524bdf
Submitter: Zuul
Branch: master

commit d09ee49afc19ad5e752755e14ac46b9af1524bdf
Author: Mark Goddard <email address hidden>
Date: Tue Nov 5 12:06:26 2019 +0000

    Fix keystone fernet rotation for source images

    In source images, keystone-manage is installed to a virtualenv in
    /var/lib/kolla/venv. This is not in the PATH for cron jobs, which always
    use PATH=/usr/bin:/bin. This results in the following error:

    /usr/bin/fernet-rotate.sh: line 3: keystone-manage: command not found

    However this error is not typically visible, since cron logs to syslog
    and we do not configure fluentd to collect these logs.

    This change configures the PATH in the fernet-rotate.sh script for
    source images.

    Change-Id: Ib49ea586d36ae32d01b9610a48b13798db4a4cd5
    Closes-Bug: #1850711

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694270

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/694271

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/694272

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694271
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=72e8aaabff413ffaa44411ee7632a43c38fc5390
Submitter: Zuul
Branch: stable/stein

commit 72e8aaabff413ffaa44411ee7632a43c38fc5390
Author: Mark Goddard <email address hidden>
Date: Tue Nov 5 12:06:26 2019 +0000

    Fix keystone fernet rotation for source images

    In source images, keystone-manage is installed to a virtualenv in
    /var/lib/kolla/venv. This is not in the PATH for cron jobs, which always
    use PATH=/usr/bin:/bin. This results in the following error:

    /usr/bin/fernet-rotate.sh: line 3: keystone-manage: command not found

    However this error is not typically visible, since cron logs to syslog
    and we do not configure fluentd to collect these logs.

    This change configures the PATH in the fernet-rotate.sh script for
    source images.

    Change-Id: Ib49ea586d36ae32d01b9610a48b13798db4a4cd5
    Closes-Bug: #1850711
    (cherry picked from commit d09ee49afc19ad5e752755e14ac46b9af1524bdf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694270
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=7adf7b1d56060b6998ce2884a38f12446b92d606
Submitter: Zuul
Branch: stable/train

commit 7adf7b1d56060b6998ce2884a38f12446b92d606
Author: Mark Goddard <email address hidden>
Date: Tue Nov 5 12:06:26 2019 +0000

    Fix keystone fernet rotation for source images

    In source images, keystone-manage is installed to a virtualenv in
    /var/lib/kolla/venv. This is not in the PATH for cron jobs, which always
    use PATH=/usr/bin:/bin. This results in the following error:

    /usr/bin/fernet-rotate.sh: line 3: keystone-manage: command not found

    However this error is not typically visible, since cron logs to syslog
    and we do not configure fluentd to collect these logs.

    This change configures the PATH in the fernet-rotate.sh script for
    source images.

    Change-Id: Ib49ea586d36ae32d01b9610a48b13798db4a4cd5
    Closes-Bug: #1850711
    (cherry picked from commit d09ee49afc19ad5e752755e14ac46b9af1524bdf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694272
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=006f7944b6e5a27c6404af1513a60b12ebe82e71
Submitter: Zuul
Branch: stable/rocky

commit 006f7944b6e5a27c6404af1513a60b12ebe82e71
Author: Mark Goddard <email address hidden>
Date: Tue Nov 5 12:06:26 2019 +0000

    Fix keystone fernet rotation for source images

    In source images, keystone-manage is installed to a virtualenv in
    /var/lib/kolla/venv. This is not in the PATH for cron jobs, which always
    use PATH=/usr/bin:/bin. This results in the following error:

    /usr/bin/fernet-rotate.sh: line 3: keystone-manage: command not found

    However this error is not typically visible, since cron logs to syslog
    and we do not configure fluentd to collect these logs.

    This change configures the PATH in the fernet-rotate.sh script for
    source images.

    Change-Id: Ib49ea586d36ae32d01b9610a48b13798db4a4cd5
    Closes-Bug: #1850711
    (cherry picked from commit d09ee49afc19ad5e752755e14ac46b9af1524bdf)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.