Migrating a volume, which is in-use, between different cinder backends triggers the error novaclient.exceptions.EndpointNotFound

[KC Bi]

While migrate a volume, which is attached to an instance, from one backend to another will trigger an erros as below:

2017-02-17 01:03:09.676 1540246 INFO cinder.volume.manager [req-979bfe01-9932-46cd-ba17-a97551550b87 71fa29ce13fb4e6fa78ac620673278c9 228b4b7366df44279a84ac71820598de - default default] Created volume successfully.
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager [req-979bfe01-9932-46cd-ba17-a97551550b87 71fa29ce13fb4e6fa78ac620673278c9 228b4b7366df44279a84ac71820598de - default default] Failed to copy volume 86974af2-8bf2-4065-a474-565a6363d7a8 to 9fa07149-f395-43bf-adb2-d9568c9174c3
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager Traceback (most recent call last):
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 1787, in _migrate_volume_generic
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager new_volume.id)
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 178, in update_server_volume
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager nova = novaclient(context, admin_endpoint=True, privileged_user=True)
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 136, in novaclient
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager **region_filter)
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/novaclient/service_catalog.py", line 84, in url_for
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager raise novaclient.exceptions.EndpointNotFound()
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager EndpointNotFound
2017-02-17 01:03:13.512 1540270 ERROR cinder.volume.manager
2017-02-17 01:03:13.515 1540270 DEBUG oslo_messaging._drivers.amqpdriver [req-979bfe01-9932-46cd-ba17-a97551550b87 71fa29ce13fb4e6fa78ac620673278c9 228b4b7366df44279a84ac71820598de - default default] CAST unique_id: 6d35fc3375a24c079d8c011d76a89ffe exchange 'cinder' topic 'cinder-volume' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448

[KC Bi (bikecheng)]

Retype in-use volume also triggers the same issue. This has been verified with Canonical Ocata release.

[Martins Jakubovics (martins-k)]

Same issue with newton 9.1.2.

[git.user (git.user)]

Looks like something is broken badly in authentication schema of cinder in this case.
I dug into a bit and i see:

1. if nova_catalog_info/nova_catalog_admin_info is not set then _NovaClientServiceCatalog->service_name is "Compute Service" (which should be 'nova' I suspect), it's here -> cinder/compute/nova.py +111 -> which leads to EndpointNotFound

2. With nova_catalog_admin_info = compute:nova:adminURL I stumble upon a next issue:

ERROR cinder.volume.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 192, in _do_create_plugin
ERROR cinder.volume.manager raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
ERROR cinder.volume.manager DiscoveryFailure: Could not determine a suitable URL for the plugin

It's because keystoneauth1/discover.py +40 -> get_version_data get nova-api url for authentication (http://controller:8774/v2.1 in my case).
It seems we need somebody with bettre understanding of thie code :(

[git.user (git.user)]

Well, then tried to set nova_endpoint_admin_template to http://controller00.ostack.fresh:35357/v3 just to see what will happen and it was very close :) The only problem (hmm... at least I think it is) I see my uuid in authentication request instead of name of user:
Member Key: identity
    Object
       Member Key: password
          Object
             Member Key: user
                 Object
..................
                     Member Key: name
                     String value: d........c30494 <- uuid
                     Key: name