test_volume_migrate_attached fails because of policy os_compute_api:os-volumes-attachments:update 403

[Matt Riedemann]

This is in a non-voting job in the nova experimental queue, but it's testing a boot from volume scenario where the volume is retyped, which triggers a volume migration and swap volume operation on the nova side.

http://logs.openstack.org/45/470645/5/experimental/gate-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend-ubuntu-xenial-nv/0b421c5/console.html#_2017-06-06_02_21_43_757566

That's failing because of a policy check:

http://logs.openstack.org/45/470645/5/experimental/gate-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend-ubuntu-xenial-nv/0b421c5/logs/screen-c-vol.txt.gz?level=TRACE#_Jun_06_02_18_21_110268

Jun 06 02:18:21.110268 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager [None req-8fa9a3f4-8a63-4bab-9631-453e54bb06d1 test_creds-1578445804 None] Failed to copy volume af82b615-3c15-4069-9573-70700c452713 to 6af6a035-55e7-41e4-8b0b-c26d86d94c1f: Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-fd23d488-cb10-4728-850f-6178c4704a8d)
Jun 06 02:18:21.110354 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager Traceback (most recent call last):
Jun 06 02:18:21.110428 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1957, in _migrate_volume_generic
Jun 06 02:18:21.110516 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager new_volume.id)
Jun 06 02:18:21.110592 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/opt/stack/new/cinder/cinder/compute/nova.py", line 160, in update_server_volume
Jun 06 02:18:21.110698 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager new_volume_id)
Jun 06 02:18:21.110769 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
Jun 06 02:18:21.110838 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager body, "volumeAttachment")
Jun 06 02:18:21.110905 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 375, in _update
Jun 06 02:18:21.110973 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager resp, body = self.api.client.put(url, body=body)
Jun 06 02:18:21.111065 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 232, in put
Jun 06 02:18:21.111146 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager return self.request(url, 'PUT', **kwargs)
Jun 06 02:18:21.111220 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 80, in request
Jun 06 02:18:21.111290 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager raise exceptions.from_response(resp, body, url, method)
Jun 06 02:18:21.111357 ubuntu-xenial-2-node-osic-cloud1-s3500-9149309 cinder-volume[30588]: ERROR cinder.volume.manager Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-fd23d488-cb10-4728-850f-6178c4704a8d)

The policy is by default admin-only:

https://github.com/openstack/nova/blob/master/nova/policies/volumes_attachments.py#L54

It looks like the Tempest test might be creating the volume types with admin credentials, but doing everything else as non-admin, which is probably why it fails.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/474789

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474789
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=1df940ebdbf13c6fb80d73be4a6a61c323350f26
Submitter: Jenkins
Branch: master

commit 1df940ebdbf13c6fb80d73be4a6a61c323350f26
Author: Matt Riedemann <email address hidden>
Date: Thu Jun 15 17:49:06 2017 -0400

    Fix test_volume_migrate_attached to retype as admin

    By default, Cinder policy will allow a non-admin owner
    of a volume to retype a volume. That triggers a call from
    Cinder to Nova to swap the volume in the guest. The Nova
    swap volume API, however, is admin-only by default. So this
    test fails with default policy because Cinder gets a 403
    response from Nova when trying to swap the volume on the
    Nova side.

    This fixes the problem by using the admin client for initiating
    the retype.

    Change-Id: I17e6603c588e3efddcf44ca4e7af6e352490e689
    Closes-Bug: #1698224

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 16.1.0 release.