Cinder

Unable to migrate volume if is attached to a instance and you a domain environment

Bug #1795997 reported by Jose Phillips on 2018-10-03

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Undecided	Unassigned

Bug Description

its imposible to migrate a volume to another storage backend because cinder is trying to use Nova v2 keystone authentication and is impossible to authenticate because no domain is send.

Volumes that are not attached to instance works great.

ARNING Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

also i attach the full stacktrace

2018-10-03 22:05:41.511 31747 DEBUG oslo_db.sqlalchemy.engines [req-2e9514bd-4669-4175-99cc-430d2404dc76 cb447635323b4f5a9742347d9320ddb7 422b299d0ab44bfba1c2d9d9f671bcc0 - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:290
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server [req-2e9514bd-4669-4175-99cc-430d2404dc76 cb447635323b4f5a9742347d9320ddb7 422b299d0ab44bfba1c2d9d9f671bcc0 - - -] Exception during message handling: BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-ed02c1c3-8905-4c4f-bee1-4789596bf6a5)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 2411, in migrate_volume
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server volume.save()
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 2404, in migrate_volume
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 2166, in _migrate_volume_generic
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server new_volume)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 2159, in _migrate_volume_generic
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server new_volume.id)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/compute/nova.py", line 180, in update_server_volume
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server new_volume_id)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 97, in update_server_volume
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server body, "volumeAttachment")
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 375, in _update
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 313, in put
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 77, in request
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server token = self.get_token(session)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 177, in get_auth_ref
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server authenticated=False, log=False, **rkwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-ed02c1c3-8905-4c4f-bee1-4789596bf6a5)
2018-10-03 22:05:41.530 31747 ERROR oslo_messaging.rpc.server
2018-10-03 22:05:41.562 31753 DEBUG cinder.volume.drivers.netapp.dataontap.nfs_cmode [req-2e9514bd-4669-4175-99cc-430d2404dc76 cb447635323b4f5a9742347d9320ddb7 422b299d0ab44bfba1c2d9d9f671bcc0 - - -] Deleting backing file for volume 04ad3597-d3d3-44b7-ac90-e49e20fa5ed3. _delete_backing_file_for_volume /usr/lib/python2.7/dist-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py:424

See original description

Jose Phillips (jose-phillips) on 2018-10-03

description:

updated

Revision history for this message

Dou Rui Yuan (rydou) wrote on 2019-01-15:

You should add domain config in keystone_authtoken section of cinder.conf like：

[keystone_authtoken]
project_domain_name = default
user_domain_name = default

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.