Volume retype fails when migration occurs

[Mohammed Naser]

While running Tempest internally, I've discovered the following test was failing:

tempest.scenario.test_volume_migrate_attached.TestVolumeMigrateRetypeAttached.test_volume_migrate_attached[compute,id-deadd2c2-beef-4dce-98be-f86765ff311b,slow,volume]

Upon further debugging, I've discovered that cinder-volume has the following traceback:

2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server [req-7b3d3148-cdbb-485d-9dd4-3ff7a24e59ad 7088d6afa8a44ef386333b30ff6d62e0 ef87e48ff4604c6aabdead43ca5cb6ca - default default] Exception during message handling: Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-d5fb2992-48ba-4459-a214-f2cceb8bb0ac)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2745, in retype
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_reservations, status_update)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2741, in retype
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_type_id=new_type_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2348, in migrate_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server volume.save()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2341, in migrate_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2107, in _migrate_volume_generic
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2100, in _migrate_volume_generic
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume.id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/compute/nova.py", line 217, in update_server_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 97, in update_server_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server body, "volumeAttachment")
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/base.py", line 375, in _update
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 297, in put
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body, url, method)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-d5fb2992-48ba-4459-a214-f2cceb8bb0ac)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server
2017-12-29 20:38:01.661 32585 INFO cinder.volume.manager [req-7b3d3148-cdbb-485d-9dd4-3ff7a24e59ad 7088d6afa8a44ef386333b30ff6d62e0 ef87e48ff4604c6aabdead43ca5cb6ca - default default] Deleted volume successfully.

Upon further checks, I've found out that during the retype process, it can fallback to migrating the volume in the event that it is not able to retype. In the migrate process, the notification of swapping the volume is sent with the same auth credentials of the user that requested this (which isn't an admin) and it fails.

I'm not sure if Nova's policy should be adjusted for this, or if Cinder should send the request with elevated context, but I believe that's where the issue stands.

[Ghanshyam Mann (ghanshyammann)]

yes, with migration_policy as 'on-demand'(this is what test do) it has to be admin user. This is fixed in your patch - https://review.openstack.org/#/c/530508/

Also i am much interested to test retype with migration_policy as 'never' so that we can check the auth behavior in retype operation. That should be success with non admin user. Let's add that test also.

[Ghanshyam Mann (ghanshyammann)]

++ tempest in this bug to add a test the retype with 'never' migration policy.

[Eric Harney (eharney)]

This may be related to bug 1698224.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/530508
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=65e4f9b9d1f0c61636895cc133062bf0a6263848
Submitter: Zuul
Branch: master

commit 65e4f9b9d1f0c61636895cc133062bf0a6263848
Author: Mohammed Naser <email address hidden>
Date: Fri Dec 29 21:22:24 2017 +0000

    Revert "Use a non admin privileges for retyping a volume"

    This reverts commit f5869b4557855af9450501c65e03bd16fd269036.

    When retyping a volume, it is possible that Cinder falls back to
    migrating a volume in order to complete the retype which is admin-only
    operation.

    This will result in a failure because Cinder will attempt to contact
    Nova using the credentials of the user which are not elevated, with
    Nova refusing to do the volume swap.

    Therefore, in order to effectively test this, we should stick to using
    the admin API as a retype can require a migration which is an
    admin-only operation.

    Related-Bug: #1740544
    Change-Id: I3e4853146bb6e2a62205ffe690da081229215f54

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/572287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572287
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=a3a67d476132fe5f55020cf2dc1f672c17be6eb3
Submitter: Zuul
Branch: master

commit a3a67d476132fe5f55020cf2dc1f672c17be6eb3
Author: whoami-rajat <email address hidden>
Date: Tue Jun 5 07:44:35 2018 +0000

    Adding test case for migration policy 'never'

    There doesn't exist any test case in tempest for retype
    volume without migration, so adding new test for it.

    Related-Bug: #1740544

    Change-Id: I58c348b954a679bc809906b2c73df9b1ab4459c5

[Martin Kopec (mkopec)]

Unassigning due to a long inactivity.

[Martin Kopec (mkopec)]

This was reported on tempest's side due to a missing test case - a new test case has been added by https://review.opendev.org/c/openstack/tempest/+/572287/ . I'm gonna close this as Fix Released for Tempest. If you feel like this needs to be addressed in Tempest further, please comment here and we'll reopen the issue (or fill a new one).