Microstack add-compute fails with expires_at error

[Mark Beierl]

Repeatable procedure on clean server:
Install snap
Initialise microstack
Attempt to add compute generates error.

Transcript of my attempt.

$ sudo snap install microstack --beta --devmode
microstack (beta) ussuri from Canonical✓ installed

$ sudo microstack init --auto --control
2020-11-05 13:05:52,936 - microstack_init - INFO - Configuring clustering ...
2020-11-05 13:05:52,995 - microstack_init - INFO - Setting up as a control node.
2020-11-05 13:05:55,209 - microstack_init - INFO - Configuring networking ...
2020-11-05 13:05:59,278 - microstack_init - INFO - Opening horizon dashboard up to *
2020-11-05 13:06:00,031 - microstack_init - INFO - Waiting for RabbitMQ to start ...
Waiting for 192.168.12.132:5672
2020-11-05 13:06:06,356 - microstack_init - INFO - RabbitMQ started!
2020-11-05 13:06:06,356 - microstack_init - INFO - Configuring RabbitMQ ...
2020-11-05 13:06:07,162 - microstack_init - INFO - RabbitMQ Configured!
2020-11-05 13:06:07,176 - microstack_init - INFO - Waiting for MySQL server to start ...
Waiting for 192.168.12.132:3306
2020-11-05 13:06:07,182 - microstack_init - INFO - Mysql server started! Creating databases ...
2020-11-05 13:06:08,158 - microstack_init - INFO - Configuring Keystone Fernet Keys ...
2020-11-05 13:06:24,418 - microstack_init - INFO - Bootstrapping Keystone ...
2020-11-05 13:06:31,663 - microstack_init - INFO - Creating service project ...
2020-11-05 13:06:36,199 - microstack_init - INFO - Keystone configured!
2020-11-05 13:06:36,214 - microstack_init - INFO - Configuring the Placement service...
2020-11-05 13:06:52,250 - microstack_init - INFO - Running Placement DB migrations...
2020-11-05 13:06:59,812 - microstack_init - INFO - Configuring nova control plane services ...
2020-11-05 13:07:08,796 - microstack_init - INFO - Running Nova API DB migrations (this may take a lot of time)...
2020-11-05 13:07:26,968 - microstack_init - INFO - Running Nova DB migrations (this may take a lot of time)...
Waiting for 192.168.12.132:8774
2020-11-05 13:08:05,926 - microstack_init - INFO - Creating default flavors...
2020-11-05 13:08:26,834 - microstack_init - INFO - Configuring nova compute hypervisor ...
2020-11-05 13:08:26,880 - microstack_init - INFO - Configuring the Spice HTML5 console service...
2020-11-05 13:08:26,926 - microstack_init - INFO - Configuring Neutron
Waiting for 192.168.12.132:9696
2020-11-05 13:09:31,508 - microstack_init - INFO - Configuring Glance ...
Waiting for 192.168.12.132:9292
2020-11-05 13:10:02,135 - microstack_init - INFO - Adding cirros image ...
2020-11-05 13:10:04,167 - microstack_init - INFO - Creating security group rules ...
2020-11-05 13:10:12,488 - microstack_init - INFO - Configuring the Cinder services...
2020-11-05 13:10:58,224 - microstack_init - INFO - Running Cinder DB migrations...
2020-11-05 13:11:08,861 - microstack_init - INFO - restarting libvirt and virtlogd ...
2020-11-05 13:11:38,969 - microstack_init - INFO - Complete. Marked microstack as initialized!

$ sudo microstack add-compute
Traceback (most recent call last):
  File "/snap/microstack/217/bin/microstack", line 11, in <module>
    load_entry_point('microstack==0.0.1', 'console_scripts', 'microstack')()
  File "/snap/microstack/217/lib/python3.8/site-packages/microstack/main.py", line 44, in main
    cmd()
  File "/snap/microstack/217/lib/python3.8/site-packages/cluster/add_compute.py", line 118, in main
    add_compute()
  File "/snap/microstack/217/lib/python3.8/site-packages/cluster/add_compute.py", line 88, in add_compute
    app_cred = _create_credential()
  File "/snap/microstack/217/lib/python3.8/site-packages/cluster/add_compute.py", line 57, in _create_credential
    return keystone_client.application_credentials.create(
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneclient/v3/application_credentials.py", line 98, in create
    return super(ApplicationCredentialManager, self).create(
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneclient/base.py", line 86, in func
    return f(*args, **new_kwargs)
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneclient/base.py", line 383, in create
    return self._post(
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneclient/base.py", line 192, in _post
    resp, body = self.client.post(url, body=body, **kwargs)
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 392, in post
    return self.request(url, 'POST', **kwargs)
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 545, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/snap/microstack/217/lib/python3.8/site-packages/keystoneauth1/session.py", line 968, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: The 'expires_at' must not be before now. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-5981e760-8bf2-4131-b0ee-1c27e69d22d9)

[Billy Olsen (billy-olsen)]

I can recreate this on my machine. It appears to be due to not passing an iso timestamp with timezone information in the keystone authtoken request (used for clustering).

A local rebuild of microstack with setting the requested expiration timestamp to UTC resolves the problem.

[Billy Olsen (billy-olsen)]

Merge proposal - https://review.opendev.org/#/c/761697/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/761697
Committed: https://git.openstack.org/cgit/x/microstack/commit/?id=d7f3c1229f6be56e4778f313fb58f6aa69a5c361
Submitter: Zuul
Branch: master

commit d7f3c1229f6be56e4778f313fb58f6aa69a5c361
Author: Billy Olsen <email address hidden>
Date: Thu Nov 5 20:42:03 2020 -0700

    Use UTC for expiration date of tokens

    Keystone assumes UTC for expires_at dates when generating auth
    tokens, so set the the expires_at to UTC timezone before making
    the request.

    Change-Id: I55cb6ccf7a8cf79057d5699372ecd27bf936643f
    Closes-Bug: #1903208