keystone (stein), python3, and postgresql: hex in database

[David Orman]

When using Keystone (Stein) with Python3 and PostgreSQL, and creating new credentials, we end up with hex in the database for the encrypted_blob in the credential. This causes Keystone to start throwing errors, as it cannot decrypt the encrypted_blob. Adding debugging code to various portions of Keystone narrowed down the location of this problem.

Relevant information, and examples will follow.

Here is the error we were seeing when we'd create a new credential:

2019-06-21 15:18:10.189 21487 ERROR keystone.credential.providers.fernet.core [req-ff10ee5f-c009-487a-8091-90a65b55eb80 STUFF STUFF - STUFF STUFF] Credential could not be decrypted. Please contact the administrator: cryptography.fernet.InvalidToken

After doing a lot of tracing, we ended up mutating: https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L41

    @sql.handle_conflicts(conflict_type='credential')
    def create_credential(self, credential_id, credential):
        testout = ("cred in sql function: ", credential['encrypted_blob'])
        LOG.error(testout)
        testout = ("cred in sql function type: ", type(credential['encrypted_blob']))
        LOG.error(testout)
        with sql.session_for_write() as session:
            ref = CredentialModel.from_dict(credential)
            testout2 = ("cred in sql function model: ", ref['encrypted_blob'])
            LOG.error(testout2)
            testout3 = ("cred in sql function model type: ", type(ref['encrypted_blob']))
            LOG.error(testout3)
            session.add(ref)
            return ref.to_dict()

Here is what the output looks like when generating a credential:

2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function type: ', <class 'bytes'>)
2019-06-21 15:18:10.091 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=')
2019-06-21 15:18:10.092 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model type: ', <class 'bytes'>)

This is what it looks like in the database:

-[ RECORD 6 ]--+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | IDWASHERE
user_id | USERIDWASHERE
project_id | PROJECTIDWASHERE
type | ec2
extra | {}
key_hash | KEYHASHWASHERE
encrypted_blob | \x67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d

If you take that encrypted_blob, and decode it:

>>> import codecs
>>> codecs.decode("67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d", "hex")
b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs='
>>>

Database and database server info:

Name | keystone
Owner | postgres
Encoding | UTF8
Collate | en_US.UTF-8
Ctype | en_US.UTF-8

ii postgresql-9.5 9.5.17-0ubuntu0.16.04.1 amd64 object-relational SQL database, version 9.5 server
ii postgresql-client-9.5 9.5.17-0ubuntu0.16.04.1 amd64 front-end programs for PostgreSQL 9.5
ii postgresql-client-common 173ubuntu0.2 all manager for multiple PostgreSQL client versions
ii postgresql-common 173ubuntu0.2 all PostgreSQL database-cluster manager
ii postgresql-contrib-9.5 9.5.17-0ubuntu0.16.04.1 amd64 additional facilities for PostgreSQL

postgres@openstack-db01:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

Keystone and associated oslo.db/sqlalchemy/psycopg information:

ii python3-keystone 2:15.0.0-0ubuntu1~cloud0 all OpenStack identity service - Python 3 library
ii python3-keystoneauth1 3.13.1-0ubuntu1~cloud0 all authentication library for OpenStack Identity - Python 3.x
ii python3-keystoneclient 1:3.19.0-0ubuntu1~cloud0 all client library for the OpenStack Keystone API - Python 3.x
ii python3-keystonemiddleware 6.0.0-0ubuntu1~cloud0 all Middleware for OpenStack Identity (Keystone) - Python 3.x

ii python3-oslo.db 4.44.0-0ubuntu1~cloud0 all database connectivity to the different backends and helper utils - Python 3.x

ii python3-sqlalchemy 1.2.15+ds1-1~cloud0 all SQL toolkit and Object Relational Mapper for Python 3
ii python3-sqlalchemy-ext 1.2.15+ds1-1~cloud0 amd64 SQL toolkit and Object Relational Mapper for Python3 - C extension

ii python3-psycopg2 2.7.4-1 amd64 Python 3 module for PostgreSQL

root@keystone01:/usr/lib/python3/dist-packages# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

Using the exact same database server, running the Rocky version of Keystone with Python2, we do not see this problem. Keystone is being installed using the Ubuntu Cloud Archive: https://wiki.ubuntu.com/OpenStack/CloudArchive

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/681736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681736
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=a4be0cb9e842e4eca2023189b19113be76f3b4d0
Submitter: Zuul
Branch: master

commit a4be0cb9e842e4eca2023189b19113be76f3b4d0
Author: Ralf Haferkamp <email address hidden>
Date: Thu Sep 12 14:44:08 2019 +0200

    Fix PostgreSQL specifc issue with credentials encoding

    Decode the encrypted credential value to a string value before handing
    it over the database, when running under Python 3.x. Otherwise the
    underlying database driver (e.g. psycopg2) might treat it as binary
    data.

    Change-Id: I87425b54f471e66a9ab3974ab46c4b7f3838b962
    Closes-Bug: #1833739

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 16.0.0.0rc1 release candidate.