Impossible to rename the Default domain id to the string 'default.'

[Marcelo Subtil Marcal]

Openstack version = Rocky

When changing the 'default_domain_id' variable to the string 'default' and changing all references for this variable in the keystone database we get the following error in keystone.log:

(keystone.common.wsgi): 2020-01-15 14:16:37,869 ERROR badly formed hexadecimal UUID string
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
    result = method(req, **params)
  File "/usr/lib/python3/dist-packages/keystone/auth/controllers.py", line 102, in authenticate_for_token
    app_cred_id=app_cred_id, parent_audit_id=token_audit_id)
  File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
    __ret_val = __f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystone/token/provider.py", line 251, in issue_token
    token_id, issued_at = self.driver.generate_id_and_issued_at(token)
  File "/usr/lib/python3/dist-packages/keystone/token/providers/fernet/core.py", line 61, in generate_id_and_issued_at
    app_cred_id=token.application_credential_id
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 159, in create_token
    protocol_id, access_token_id, app_cred_id
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 444, in assemble
    b_domain_id = cls.convert_uuid_hex_to_bytes(domain_id)
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 290, in convert_uuid_hex_to_bytes
    uuid_obj = uuid.UUID(uuid_string)
  File "/usr/lib/python3.6/uuid.py", line 140, in __init__
    raise ValueError('badly formed hexadecimal UUID string')
ValueError: badly formed hexadecimal UUID string
(keystone.common.wsgi): 2020-01-15 14:16:38,908 WARNING You are not authorized to perform the requested action: identity:get_domain.
(keystone.common.wsgi): 2020-01-15 14:16:39,058 WARNING You are not authorized to perform the requested action: identity:get_domain.
(keystone.common.wsgi): 2020-01-15 14:16:50,838 WARNING You are not authorized to perform the requested action: identity:list_projects.
(keystone.common.wsgi): 2020-01-15 14:16:54,086 WARNING You are not authorized to perform the requested action: identity:list_projects.

This change is needed to integrate keystone to ICO (IBM Cloud Orchestrator)

[Marcelo Subtil Marcal (msmarcal)]

Subscribed field-critical

[James Page (james-page)]

Snip from assemble method:

        try:
            b_domain_id = cls.convert_uuid_hex_to_bytes(domain_id)
        except ValueError:
            # the default domain ID is configurable, and probably isn't a UUID
            if domain_id == CONF.identity.default_domain_id:
                b_domain_id = domain_id
            else:
                raise

Note explicit catch of ValueError to deal with configurable default domain ID and not being a UUID.

[James Page (james-page)]

Please can you confirm that keystone has the configuration set correctly in the keystone.conf

[identity]
default_domain_id = default

Its not sufficient to just change values in the DB

[James Page (james-page)]

Although looking at the code I don't see how that could be anything other than the UUID of the default domain as it generated by the charm and written out as part of the template for keystone.conf for all releases.

[James Page (james-page)]

if domain_id != to the default_domain_id in the configuration file the exception just gets re-raised

[James Page (james-page)]

FTR charm has written the UUID to the configuration file for the last 3 years:

  https://opendev.org/openstack/charm-keystone/commit/ccf15398

[Marcelo Subtil Marcal (msmarcal)]

The keystone configuration seems to be correct:

ubuntu@faye:~$ for i in $(seq 0 2); do juju ssh keystone/$i sudo grep default_domain_id /etc/keystone/keystone.conf; done
default_domain_id = default
Connection to 192.168.210.131 closed.
default_domain_id = default
Connection to 10.0.1.26 closed.
default_domain_id = default
Connection to 10.0.1.23 closed.

[Frode Nordahl (fnordahl)]

The domain named ``default`` having an literal ID of ``default`` is for backwards compatibility.

Prior to Queens (when the v2.0 API was still there) the current bootstrap method for the Keystone charm would give you a default domain with ``default`` as ID. On Queens onward the current bootstrap method will always give you a UUID as ID for the default domain.

We can restore the backwards compatibility by moving to use the ``keystone-manage bootstrap`` command to bootstrap Keystone and drop the use of the admin_token.

[Marcelo Subtil Marcal (msmarcal)]

Just one more detail. We'll need to do this change also in Queens which is the current LTS.

[Frode Nordahl (fnordahl)]

Yes, the suggested approach will support this all the way back to Mitaka: https://review.opendev.org/#/c/712040/

[Alex Kavanagh (ajkavanagh)]

Manual testing of mojo specs of the change in keystone is successful!:

# stable_to_next

$ ./go-mojo.sh bionic specs/full_stack/stable_to_next/queens
2020-03-13 12:19:57 [INFO] Using pre-existing container class containerless
2020-03-13 12:19:57 [INFO] Rsyncing /home/ubuntu/openstack-charmers/openstack-mojo-specs to /srv/mojo/openstack/bionic/_20201913_121956/spec
2020-03-13 12:20:00 [INFO] Retrieve the spec's manifest
2020-03-13 12:20:00 [INFO] Manifest comment:

...

Collect the development charms
#############################################################################

2020-03-13 12:56:27 [DEBUG] ### Running phase collect with options {'config': 'collect-next'} ###
2020-03-13 12:56:27 [INFO] Building resource tree
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/tempest-92 from charm store to tempest
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/neutron-openvswitch-396 from charm store to neutron-openvswitch
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/ceph-mon-412 from charm store to ceph-mon
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/swift-proxy-189 from charm store to swift-proxy
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/neutron-api-438 from charm store to neutron-api
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/openstack-dashboard-468 from charm store to openstack-dashboard
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/vault-84 from charm store to vault
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/nova-cell-controller-34 from charm store to nova-cell-controller
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/heat-345 from charm store to heat
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/cinder-ceph-312 from charm store to cinder-ceph
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/ceph-osd-440 from charm store to ceph-osd
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/cinder-405 from charm store to cinder
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/percona-cluster-364 from charm store to mysql
2020-03-13 12:56:42 [INFO] Downloading cs:~thedac/mongodb-0 from charm store to mongodb
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/nova-compute-488 from charm store to nova-compute
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/swift-storage-352 from charm store to swift-storage-z2
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/neutron-gateway-410 from charm store to neutron-gateway
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/nova-cloud-controller-475 from charm store to nova-cloud-controller
2020-03-13 12:56:42 [INFO] Downloading cs:~openstack-charmers-next/ceilometer-agent-315 from charm store to ceilometer-agent
2020-03-13 12:56:43 [INFO] Downloading cs:~openstack-charmers-next/swift-storage-352 from charm store to swift-storage-z3
2020-03-13 12:56:43 [INFO] Downloading cs:~openstack-charmers-next/gnocchi-87 from charm store to gnocchi
2020-03-13 12:56:43 [INFO] Downloading cs:~o...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/712040
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=0a02c30fe5f4650235519897b71588ae22fa0971
Submitter: Zuul
Branch: master

commit 0a02c30fe5f4650235519897b71588ae22fa0971
Author: Frode Nordahl <email address hidden>
Date: Mon Mar 9 15:06:09 2020 +0100

    Replace use of admin_token with Keystone bootstrap

    Stop the use of the admin_token and use the bootstrap process
    to initialize Keystone instead. Fortunately the implementation
    of the bootstrap process is both idempotent when it needs to be
    and it can be safely called on an existing deployment.

    Subsequently we can migrate by just removing the admin_token
    from the configuration and create new credentials for use by
    the charm with a call to ``keystone-manage bootstrap``.

    Remove configuration templates for versions prior to Mitaka, by
    doing this we need to move any configuration initially defined
    prior to Miataka forward to the ``templates/mitaka`` folder.

    A side effect of this migration is that newly bootstrapped
    deployments will get their ``default`` domain created with a
    literal ID of ``default``. Prior to this change third party
    software making assumptions about that being the case may have
    had issues.

    Closes-Bug: #1859844
    Closes-Bug: #1837113
    Related-Bug: #1774733
    Closes-Bug: #1648719
    Closes-Bug: #1578678
    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/191
    Change-Id: I23940720c24527ee34149f035c3bdf9ff54812c9

[Marcelo Subtil Marcal (msmarcal)]

I tested the new keystone charm, and it seems that works properly:

$ openstack domain list
+----------------------------------+----------------+---------+--------------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------+---------+--------------------+
| 752bebe6cd024ec386970a8d8fe1f0a6 | admin_domain | True | Created by Juju |
| bbc93aa69bf94198b7c751845b69f62a | service_domain | True | Created by Juju |
| default | Default | True | The default domain |
+----------------------------------+----------------+---------+--------------------+

I also had to update the ceph-radosgw charm to get the test bundle being deployed with no errors.

[James Page (james-page)]

As this is quite an invasive change I'm proposing that we don't stable backport this and wait for the 20.05 charm release for delivery into stable.