While testing HA in RDOPhase2 we do a test named "pacemaker-light-a". We stop all the systemd resources managed by Pacemaker (so just openstack-cinder-volume), then we stop overcloud's core resources (Galera, Rabbit, and Redis) and then we start openstack-cinder-volume again, checking that we don't have failed actions on the cluster.
After the recovery of the core resources, so again after verifying that we don't have failed actions on the cluster side, we start an openstack-heat template [1] that spawns an instance using a previously created volume.
We hit this problem while creating the stack:
2018-06-20 21:37:21Z [stack_test_pacemaker-light-a.instance]: CREATE_FAILED Unauthorized: resources.instance: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)",
2018-06-20 21:37:23Z [stack_test_pacemaker-light-a]: CREATE_FAILED Resource CREATE failed: Unauthorized: resources.instance: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)
The error seems to be related to the authentication layer. Looking at the heat logs [2] the full error is:
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource Traceback (most recent call last):
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 919, in _action_recorder
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource yield
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1027, in _do_action
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource step = next(subtask)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 978, in action_handler_task
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource done = check(handler_data)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 872, in check_create_complete
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource check = self.client_plugin()._check_active(server_id)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 221, in _check_active
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource server = self.fetch_server(server)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 150, in fetch_server
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource server = self.client().servers.get(server_id)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 811, in get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource return self._get("/servers/%s" % base.getid(server), "server")
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 351, in _get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource resp, body = self.api.client.get(url)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource return self.request(url, 'GET', **kwargs)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource
But there seem not to be an explicit error inside keystone log [3], so it might be related to some other component.
[1] https://github.com/openstack/tripleo-ha-utils/blob/master/roles/validate-ha/templates/validate-ha-heat-template.yaml.j2
[2] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-174/overcloud-controller-0/var/log/containers/heat/heat_api.log.txt.gz
[3] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-174/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz
Everything has changed in this bug since its last meaningful update. Raoul has moved to other positions, rdophase2 is not in our scope anymore. MOving to WOntfix