Vm recovery fails cause of wrong endpoint

[Eric Xie]

Description
===========
Got error when executing recovery of vm's failover.

Steps to reproduce
==================
* Create one notification
{
 "notification": {
  "type": "VM",
  "hostname": "host-172-23-28-162",
  "generated_time": "2017-06-13",
  "payload": {
   "instance_uuid": "2bff87d5-9b21-4046-ad1b-9f0eb7ba1805",
   "event": "IO_ERROR",
   "vir_domain_event": "IO_ERROR_REPORT"
  }
 }
}

Expected result
===============
Got error.

Actual result
=============
The status of notification is 'finished'.

Environment
===========
# ENV
devstack

# version
- devstack:
git log
commit 9234316794403e6ec422c3c9a766edbb1bae180f
Merge: 1ca22d5 bb436d3
Author: Jenkins <email address hidden>
Date: Fri Jun 9 01:55:36 2017 +0000

    Merge "Make use of Ubuntu Cloud Archive switchable"

- masakari:
$ git log
commit 00969f0bf21e5d3eda0f196dba2fd63abc7b28e9
Merge: 1e5fab1 171b4d2
Author: Jenkins <email address hidden>
Date: Tue Jun 6 05:08:01 2017 +0000

    Merge "Revert "Fix os_privileged_user_auth_url""

Logs
===========
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server [req-bc7ac122-0095-45d0-b03a-d2c6b24ffc65 nova - - - -] Exception during message handling: DiscoveryFailure: Could not determine a suitable URL for the plugin
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 218, in process_notification
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server self._process_notification(context, notification)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 214, in _process_notification
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server do_process_notification(notification)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/utils.py", line 284, in inner
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 198, in do_process_notification
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server context, notification)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 112, in _handle_notification_type_instance
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server notification.notification_uuid)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/drivers/taskflow/driver.py", line 174, in execute_instance_failure
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server flow_engine.run()
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server failures[0].reraise()
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/drivers/taskflow/instance_failure.py", line 46, in execute
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server instance = self.novaclient.get_server(context, instance_uuid)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/compute/nova.py", line 58, in wrapper
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/compute/nova.py", line 191, in get_server
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return nova.servers.get(uuid)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 799, in get
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return self._get("/servers/%s" % base.getid(server), "server")
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 351, in _get
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server resp, body = self.api.client.get(url)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 234, in get
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 82, in request
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 393, in request
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 159, in request
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return wrapped(*args, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server token = self.get_token(session)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server self._plugin = self._do_create_plugin(session)
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
Jun 13 17:28:51 localhost masakari-engine: 2017-06-13 17:28:51.094 9225 ERROR oslo_messaging.rpc.server DiscoveryFailure: Could not determine a suitable URL for the plugin

[Eric Xie (eric-xie)]

At line https://github.com/openstack/masakari/blob/master/devstack/plugin.sh#L116:
iniset $MASAKARI_CONF DEFAULT os_privileged_user_auth_url "${KEYSTONE_AUTH_PROTOCOL}://${KEYSTONE_AUTH_HOST}/identity_admin"

But the default endpoint of identity service is:
# openstack endpoint list | grep identity
| 0e14c329ab594f218309c37c1bf3df08 | RegionOne | keystone | identity | True | admin | http://172.23.28.157/identity |
| e4129ccf60204411a395e6330dcf5ca5 | RegionOne | keystone | identity | True | public | http://172.23.28.157/identity |

So modify 'identity_admin' to 'identity'.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/473782

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Xie ([e-mail]) on branch: master
Review: https://review.openstack.org/473782