masakari-engine fails with latest OpenStack installed with devstack giving EndpointNotFound exception.

[Dinesh Bhor]

When you install OpenStack with latest devstack it doesn't create adminURL for nova.
masakari-engine calls nova with defaulted 'adminURL' which is set by config option 'nova_catalog_admin_info'.

Sean Dague has submitted a patch in devstack to simplify endpoints: https://review.openstack.org/#/c/433272/1

With this change projects which were using the core services internally with adminURL got affected.
Cinder, Manila, Sahara were using nova internally with 'adminURL' so recently they have started using nova's publicURL.

Cinder: https://review.openstack.org/#/c/433289/2
Sahara: https://review.openstack.org/#/c/436567/1
Manila: https://review.openstack.org/#/c/405530/

NOTE:
This change doesn't affect ocata release.
User will get this issue only if he/she pulls the new devstack and install OpenStack with that.

masakari-engine logs:

2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 226, in put
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver return self.request(url, 'PUT', **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 74, in request
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 374, in request
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 142, in request
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver return self.session.request(url, method, **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver return wrapped(*args, **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 513, in request
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver **endpoint_filter)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 856, in get_endpoint
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver return auth.get_endpoint(self, **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 216, in get_endpoint
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver service_name=service_name)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver return wrapped(*args, **kwargs)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 228, in url_for
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver raise exceptions.EndpointNotFound(msg)
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver EndpointNotFound: adminURL endpoint for compute service not found
2017-03-01 15:56:14.851 TRACE masakari.engine.drivers.taskflow.driver
2017-03-01 15:56:14.855 DEBUG masakari.engine.drivers.taskflow.driver [req-77d48ed5-e88a-46ec-9e32-1ca670dec2f4 service None] Task
'masakari.engine.drivers.taskflow.host_failure.DisableComputeServiceTask;instance:evacuate' (f3208620-b7d1-460d-b245-e042e7a78cce)
transitioned into state 'REVERTING' from state 'FAILURE' from (pid=23676) _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:194
2017-03-01 15:56:14.857 WARNING masakari.engine.drivers.taskflow.driver [req-77d48ed5-e88a-46ec-9e32-1ca670dec2f4 service None]
Task 'masakari.engine.drivers.taskflow.host_failure.DisableComputeServiceTask;instance:evacuate' (f3208620-b7d1-460d-b245-e042e7a78cce)
transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
2017-03-01 15:56:14.858 WARNING masakari.engine.drivers.taskflow.driver [req-77d48ed5-e88a-46ec-9e32-1ca670dec2f4 service None]
Flow 'instance_evacuate_engine' (0e9be3fc-f8cb-4f77-afaa-f0ed57208f73) transitioned into state 'REVERTED' from state 'RUNNING'
2017-03-01 15:56:14.859 ERROR masakari.engine.drivers.taskflow.driver [req-77d48ed5-e88a-46ec-9e32-1ca670dec2f4 service None]
Failed to execute host failure flow for notification '35a17541-4fec-4e86-893c-3be27cd51e85'.
2017-03-01 15:56:14.859 INFO masakari.engine.manager [req-77d48ed5-e88a-46ec-9e32-1ca670dec2f4 service None]
Notification 35a17541-4fec-4e86-893c-3be27cd51e85 exits with status: error.

Masakari master:
commit 7415951c46e21297a1f8a3c878595cd399669831
Author: Kengo Takahara <email address hidden>
Date: Thu Feb 23 19:04:42 2017 +0900

    Prevent 404 error when adding reserved_host to aggregate

    When host-failure occurs, masakari-engine adds reserve_host
    to aggregate.
    However, when masakari-engine adds reserved_host,
    masakari-engine passes an aggregate_name to novaclient.
    This patch is modified so that masakari-engine passes
    aggregate_id instead of aggregate_name to novaclient.

    Change-Id: I669b19dea04c8ebb3a27a8ae746ae4c3f88d66f0
    Closes-Bug: #1667246

devstack master:

commit 192046e4abce55367ead0378b8a6037ad8f1f710
Merge: 0e1e781 52bb641
Author: Jenkins <email address hidden>
Date: Tue Feb 28 21:28:52 2017 +0000

    Merge "Use qemu-kvm-ev package on centos"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/440329

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/440329
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=3f67b04e2158856f69ec5c0aee3672c7b9933ed8
Submitter: Jenkins
Branch: master

commit 3f67b04e2158856f69ec5c0aee3672c7b9933ed8
Author: dineshbhor <email address hidden>
Date: Wed Mar 1 17:15:05 2017 +0530

    Change nova_catalog_admin_info to default to publicURL

    A recent change in devstack [1] reduced the number of endpoints
    configured by default. This breaks masakari resulting into:
    "EndpointNotFound: adminURL endpoint for compute service not found"

    This patch changes the default for the nova_catalog_admin_info to be
    the normal 'publicURL'. With 'publicURL' masakari-engine is able to
    call nova's admin restricted apis.

    [1] Ided7a65c81b3a0b56f0184847fc82e17c29a771e
    Closes-Bug: #1669280
    Change-Id: I90bc214249f8a542f2d40fca544af204e0665821