kolla

Keystone key repository missing for credentials

Series rocky
Bug #1863643

Bug #1863643 reported by Pierre Riteau on 2020-02-17

This bug affects 11 people

	Status	Importance	Assigned to
kolla
Rocky	New	Medium	Unassigned
Stein	New	Medium	Unassigned
Train	New	Medium	Unassigned
Ussuri	Triaged	Medium	Dincer Celik
kolla-ansible	Won't Fix	Medium	Unassigned
Rocky	Won't Fix	Medium	Unassigned
Stein	Won't Fix	Medium	Unassigned
Train	Won't Fix	Medium	Unassigned
Ussuri	Won't Fix	Medium	Unassigned

Bug Description

Keystone uses the [credential]/key_repository configuration option to point to a directory containing Fernet keys used to encrypt and decrypt credentials stored in the credential backend. The default value is /etc/keystone/credential-keys/

This is not configured with Kolla-Ansible. On a Stein deployment, we see periodic ERROR logs associated with it:

2020-02-17 17:00:25.263 30 ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/

Note from the docs: Fernet keys used to encrypt credentials have no relationship to Fernet keys used to encrypt Fernet tokens. Both sets of keys should be managed separately and require different rotation policies. Do not share this repository with the repository used to manage keys for Fernet tokens.

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-02-19:

Pierre, do you know what would be affected by this? Seems like it must be a less frequently used feature otherwise we'd hear about it.

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-02-19:

Perhaps it implies our credentials are not encrpyted in the DB? https://docs.openstack.org/keystone/pike/admin/identity-credential-encryption.html

Mark Goddard (mgoddard) on 2020-02-19

Changed in kolla-ansible:
importance:	Undecided → Medium

Dincer Celik (dincercelik) on 2020-02-19

Changed in kolla-ansible:
assignee:	nobody → Dincer Celik (osmanlicilegi)

Dincer Celik (dincercelik) on 2020-02-19

affects:

kolla-ansible → kolla

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-20: Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708955

Changed in kolla:
status:	New → In Progress

Revision history for this message

Christopher Wellie (well100) wrote on 2020-06-03:

I have the same problem under train

Mark Goddard (mgoddard) on 2020-06-24

Changed in kolla:
milestone:	10.0.0 → none

Revision history for this message

Christopher Wellie (well100) wrote on 2020-11-11:

When will the bug be processed?

Revision history for this message

Mark Goddard (mgoddard) wrote on 2020-11-13:

Hi Christopher. This bug has been open for some time, and although it is marked as in progress, I don't think Dincer is working on it (he may correct me). In that case, it will be processed when someone picks it up and works on it. Perhaps you are able to help?

Changed in kolla:
status:	In Progress → Triaged

Revision history for this message

Till Plüer (tplueer) wrote on 2021-09-29:

Same problem still exists in victoria

Revision history for this message

Jacolex (jacolex) wrote on 2021-11-29:

Hi, the issue still persists in Xena. I have to deploy manual scripts based on fernet-rotate.sh with oneliner

keystone-manage --config-file /etc/keystone/keystone.conf credential_rotate --keystone-user keystone --keystone-group keystone

Additionally I've added rsync just like fernet-node-sync.sh and I've modified crontab.

Revision history for this message

yule sun (syle87) wrote on 2022-09-15:

I have a issue like this when i upgrade my cluster from Queens to Rocky, I got an error like this :

2022-09-15 14:47:36.707 23 WARNING stevedore.named [-] Could not load uuid
2022-09-15 14:48:28.868 133 ERROR keystone.common.fernet_utils [-] Either [credential] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/
2022-09-15 14:48:28.869 133 ERROR keystone.common.fernet_utils [-] Either [credential] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/

And i can't access the keystone endpoint , is this a same issue?

[root@zlmanager01 keystone]# curl http://192.168.2.254:35357/v3
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
root@localhost to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Revision history for this message

Pierre Riteau (priteau) wrote on 2022-09-15:

#10

Hello yule sun (syle87),

The error you are seeing is more likely related to the first WARNING log:

2022-09-15 14:47:36.707 23 WARNING stevedore.named [-] Could not load uuid

UUID tokens were removed in Queens. You probably need to update keystone.conf to use

[token]
provider = fernet

instead of:

[token]
provider = uuid

The ERROR logs about credential keys are not fatal.

Revision history for this message

Jacolex (jacolex) wrote on 2022-09-15:

#11

Hello
My files and script templates changes:

additional lines:

/usr/local/share/kolla-ansible/ansible/roles/keystone/defaults/main.yml
[...]
  keystone-ssh:
    volumes:
      - "keystone_credential_tokens:/etc/keystone/credential-keys"
  keystone-fernet:
    volumes:
      - "keystone_credential_tokens:/etc/keystone/credential-keys"
keystone_default_volumes:
  - "{% if keystone_token_provider == 'fernet' %}keystone_credential_tokens:/etc/keystone/credential-keys{% endif %}"

additional lines in two scripts regarding to credential-keys. Also commenting: set -o errexit, set -o pipefail.

/usr/local/share/kolla-ansible/ansible/roles/keystone/templates/fernet-push.sh.j2
#set -o errexit
#set -o pipefail
[...]
/usr/bin/rsync --dry-run -az -e 'ssh -i /var/lib/keystone/.ssh/id_rsa -p {{ hostvars[host]['keystone_ssh_port'] }} -F /var/lib/keystone/.ssh/config' --delete /etc/keystone/credential-keys/ keystone@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:/etc/keystone/credential-keys
[...]
/usr/bin/rsync -az -e 'ssh -i /var/lib/keystone/.ssh/id_rsa -p {{ hostvars[host]['keystone_ssh_port'] }} -F /var/lib/keystone/.ssh/config' --delete /etc/keystone/credential-keys/ keystone@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:/etc/keystone/credential-keys
[...]

/usr/local/share/kolla-ansible/ansible/roles/keystone/templates/fernet-rotate.sh.j2
#set -o errexit
#set -o pipefail
[...]
keystone-manage --config-file /etc/keystone/keystone.conf fernet_rotate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
keystone-manage --config-file /etc/keystone/keystone.conf credential_migrate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
keystone-manage --config-file /etc/keystone/keystone.conf credential_rotate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
[...]

Revision history for this message

yule sun (syle87) wrote on 2022-09-16:

#12

    Thank you Pierre Riteau and Jacolex
    Fixed this issuse when i upgraded the /etc/kolla/keystone/keystone.conf, change the provider=uuid to provider=fernet, and then use he command reconfigure to update the keystone roles.
    Thank you both very much for your replies and answers.
    Best wishes to you.

Revision history for this message

Michal Nasiadka (mnasiadka) wrote on 2023-01-31:

#13

All versions from Rocky to Ussuri are EOL, marking as won't fix.

no longer affects:	kolla
Changed in kolla-ansible:
status:	New → Won't Fix

Revision history for this message

Jacolex (jacolex) wrote on 2023-06-27:

#14

Hello
What about new versions of kolla? Stil no support for credential keys.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.