[Queens] tokens generated with nocatalog are not usable in some requests

[Pavlo Shchelokovskyy]

NOTE - this is happening only on Queens, and possibly earlier, and is already silently fixed in Rocky as part of the major refactor of token model. I am posing this issue here so that anyone still running Queens has a reference and probably a patch to apply.

In Queens release, if I create a token using a nocatalog option:

  curl -X POST <keystone-endpoint>/v3/auth/tokens?nocatalog

and then use this token to e.g. list servers with details

  curl -X GET <nova-endpoint>/v2.1/servers/detail

I get 500 error from nova, with nova api logs containing

  ERROR nova.api.openstack EmptyCatalog: The service catalog is empty.

When repeating the same request with the same token after 5-10 minutes, the token starts to work.
Tokens generated with catalog are working as well.

AFAIU this goes down to token caching - in Queens tokens are cached/memoized with catalog - or without it if token was requested w/o catalog. Then on token validation, the token validation response takes the token - with or without catalog - from cache and returns it to caller with minimal processing - e.g. removing catalog if token validation call asked for it. It does not however ensures that the catalog is present otherwise.

This breaks some other services like Nova, that expects the catalog to be present in the request context constructed from the keystonemiddleware results. Nova needs this for example to make API requests to other services - exactly what happens in server/details call where it has to ask Neutron for some network info about instances.

After the cache is invalidated, the catalog starts to be generated for token validation response anew, and everything starts to work as expected.

[Pavlo Shchelokovskyy (pshchelo)]

patch is proposed here https://review.opendev.org/c/openstack/keystone/+/802751

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Douglas Mendizábal <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/keystone/+/802751
Reason: Abandoning unmerged stable/queens changes.

[Pavlo Shchelokovskyy (pshchelo)]

let's close this as won't fix then, since Queens is already dead