volume migration failed because EndpointNotFound

[weiweigu]

Migration an in-use volume to another volume backend host failed:
2016-03-22 09:46:57.783 3812 ERROR oslo_messaging.rpc.dispatcher [req-b77dec6f-9f1b-45ef-9b99-bf5cc3ddbb02 5650304261d74dbe8a4f7848661f95a6 55bbe36a87af48c0af04c9204a49a854 - - -] Exception during message handling:
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1622, in migrate_volume
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher self.db.volume_update(ctxt, volume_ref['id'], updates)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1611, in migrate_volume
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher new_type_id)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1424, in _migrate_volume_generic
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher new_volume['id'])
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1417, in _migrate_volume_generic
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher new_volume['id'])
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 171, in update_server_volume
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher novaclient(context).volumes.update_server_volume(server_id,
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 131, in novaclient
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher **region_filter)
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/novaclient/service_catalog.py", line 84, in url_for
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher raise novaclient.exceptions.EndpointNotFound()
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher EndpointNotFound
2016-03-22 09:46:57.783 3812 TRACE oslo_messaging.rpc.dispatcher

After analyzing code, I found the reason was the default value of 'nova_catalog_info'.
The default value of 'nova_catalog_info' is 'compute:Compute Service:publicURL' in /cinder/compute/nova.py. 'nova_catalog_info' format is: separated values of the form:<service_type>:<service_name>:<endpoint_type>.

In keystone, the compute service's name is 'nova' , not 'Compute Service'.
I think the default value of 'nova_catalog_info' should be 'compute:nova:publicURL'.

See the following code:
'nova_catalog_admin_info'
nova_opts = [
    cfg.StrOpt('nova_catalog_info',
               default='compute:Compute Service:publicURL',
               help='Match this value when searching for nova in the '
                    'service catalog. Format is: separated values of '
                    'the form: '
                    '<service_type>:<service_name>:<endpoint_type>'),
    cfg.StrOpt('nova_catalog_admin_info',
               default='compute:Compute Service:adminURL',
               help='Same as nova_catalog_info, but for admin endpoint.'),
    ...]

See keystone cli info:
[root@2C5_19_CG1 cinder(keystone_admin)]# keystone service-list
+----------------------------------+------------+----------------+----------------------------------+
| id | name | type | description |
+----------------------------------+------------+----------------+----------------------------------+
| 69ef68182b82410fbf3030552455e2e0 | cinder | volume | Cinder Service |
| 7f86b30b810f41a1a37e4b078fe6157b | nova | compute | Openstack Compute Service |

And another config 'nova_catalog_admin_info' has the same problem.

[Gorka Eguileor (gorka)]

That would be a backward incompatible change that could break existing systems on upgrade...

[weiweigu (gu-weiwei)]

Thank you for your reply.
In Icehouse version, the compute service's name is "nova". It was "Compute Service" in the earlier version?