When I perform a cinder retype from a Solidfire volume type to a ONTAP volume type, I see
a "No host supplied" error.
Command run: cinder retype --migration-policy on-demand solidfire_vol ontap-nfs
Releases affected: Yoga, Zed
Environments:
1. Running packstack / kolla-ansible deployments
2. A Stein controller (using a very similar cinder.conf and connected to the same storage arrays)
does not have this issue. cinder retype completes without issue.
3. My volume type only specifies the volume_backend_name. There are no extra_specs
4. My SolidFire volume backend is pretty simple:
[solidfire]
volume_driver = cinder.volume.drivers.solidfire.SolidFireDriver
san_ip = xxxx
san_login = xxxx
san_password = xxxx
sf_svip=xxxxx
sf_account_prefix = ''
volume_backend_name=solidfire
In the cinder volume.log the exception I see is:
2023-04-25 13:37:19.428 32 DEBUG cinder.volume.manager [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368a4472a9c4
a48f1fdce8057 - - - -] Issue driver.migrate_volume. migrate_volume /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py:2671
2023-04-25 13:37:19.429 32 INFO cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368
a4472a9c4a48f1fdce8057 - - - -] Migrate volume 6b2fc7f8-d0be-4641-8d5a-fd7b558b89aa to xxxx@ontap-nfs#xxxx:/cinder_ha_vol.
2023-04-25 13:37:19.431 32 INFO cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368
a4472a9c4a48f1fdce8057 - - - -] Source and destination clusters are different. A cluster migration will be performed.
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Active cluster: [solidfire_array], Destination: [] migrate_volume /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2445
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Start migrating volume [volume-6b2fc7f8-d0be-4641-8d5a-fd7b558b89aa] to cluster [xxxx@ontap-nfs#xxxx:/cinder_ha_vol] _do_intercluster_volume_migration /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2376
..
2023-04-25 13:37:19.432 32 DEBUG cinder.volume.drivers.solidfire [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da3238436
8a4472a9c4a48f1fdce8057 - - - -] Destination cluster mvip is [] _do_intercluster_volume_migration /var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py:2381
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server [req-88faa23e-b5b8-4636-b1fe-003291e2594a req-7db290b9-f0b6-4594-be53-9c5276904af7 040e9a0e099846d898ec2675de2054c8 da32384368a4472
a9c4a48f1fdce8057 - - - -] Exception during message handling: requests.exceptions.InvalidURL: Invalid URL 'https://:443/json-rpc/1.0/': No host supplied
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 3143, in retype
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise self.value
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 3140, in retype
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.migrate_volume(context, volume, host,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 2693, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise self.value
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/manager.py", line 2672, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server moved, model_update = self.driver.migrate_volume(ctxt,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 2450, in migrate_volume
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server updates = self._do_intercluster_volume_migration(volume, host,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 2383, in _do_intercluster_volume
_migration
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server dst_cluster_ref = self._create_cluster_reference(dst_endpoint)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 544, in _create_cluster_referenc
e
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server cluster_info = (self._issue_api_request(
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 182, in func_retry
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/cinder/volume/drivers/solidfire.py", line 678, in _issue_api_request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server req = requests.post(url,
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 115, in post
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return request("post", url, data=data, json=json, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server return session.request(method=method, url=url, **kwargs)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 573, in request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server prep = self.prepare_request(req)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 484, in prepare_request
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server p.prepare(
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/models.py", line 368, in prepare
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server self.prepare_url(url, params)
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/models.py", line 445, in prepare_url
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server requests.exceptions.InvalidURL: Invalid URL 'https://:443/json-rpc/1.0/': No host supplied
2023-04-25 13:37:19.465 32 ERROR oslo_messaging.rpc.server
The issue appears to be this in the workflow:
Destination cluster mvip is [] _do_intercluster_volume_migration
Looking at https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/solidfire.py, is the
SF driver only looking for destination "solidfire" addresses? And the retype is failing because
it's a ONTAP destination?
It looks like there are two types of migrations available:
1. storage assisted migration
2. host assisted migration
As I understand it, during the Victoria release storage assisted migration was added to the Solidfire driver.
I believe that is what the Solidfire driver is currently trying to do. Perform a storage assisted
migration between two different storage vendor's backends (Solidfire to ONTAP). I believe what it should
be doing is performing a host assisted migration which would be successful.
cinder migrate has the "--force-host-copy" option. However cinder retype does not have an option like this.
Unless there is a way to force a host assisted migration with cinder retype, it looks like any attempt to
retype from a Solidfire volume type to a volume type used by another storage vendor's backend will fail.