Cinder volumes are stuck when non admin user executes nova swap volume API

[Takashi Natsume]

Cinder volumes are stuck in attaching/detaching state when non admin user executes nova swap volume API.
Because cinder 'migrate_volume_completion' API can be executed by admin only in default settings of cinder policy.json.
So the default settings of cinder policy.json should be fixed.

[How to reproduce]
stack@devstack-master:/opt/devstack$ env | grep OS
(snipped...)
OS_USERNAME=demo
OS_TENANT_NAME=demo
(snipped...)
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
| 5a4102cd-9e36-480c-a148-d2a127ff704e | server1 | ACTIVE | - | Running | private=10.0.10.3, fd61:9f1e:73ec:0:f816:3eff:fe34:8b61 |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------------------+
stack@devstack-master:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| 1c768de0-949f-485f-b914-c430752e2984 | available | TEST2 | 1 | lvmdriver-1 | false | False | |
| 999a256e-aadf-41b6-88f3-1412e3462cbe | in-use | TEST1 | 1 | lvmdriver-1 | false | False | 5a4102cd-9e36-480c-a148-d2a127ff704e |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
stack@devstack-master:/opt/devstack$ nova volume-update server1 999a256e-aadf-41b6-88f3-1412e3462cbe 1c768de0-949f-485f-b914-c430752e2984
stack@devstack-master:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+
| 1c768de0-949f-485f-b914-c430752e2984 | attaching | TEST2 | 1 | lvmdriver-1 | false | False | |
| 999a256e-aadf-41b6-88f3-1412e3462cbe | detaching | TEST1 | 1 | lvmdriver-1 | false | False | 5a4102cd-9e36-480c-a148-d2a127ff704e |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+--------------------------------------+

[cinder-api.log]
2015-12-04 15:41:51.986 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] POST http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action
2015-12-04 15:41:51.986 DEBUG cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] Action body: {"os-migrate_volume_completion": {"new_volume": "1c768de0-949f-485f-b914-c430752e2984", "error": false}} from (pid=18203) get_method /opt/stack/cinder/cinder/api/openstack/wsgi.py:1093
2015-12-04 15:41:51.987 INFO cinder.api.openstack.wsgi [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] http://10.0.2.15:8776/v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action returned with HTTP 403
2015-12-04 15:41:51.989 INFO eventlet.wsgi.server [req-44d81e36-299f-4e14-9419-996a2239b1ca 181cd8bc1c004030aec893fc7de79618 722380ff5288483191cd2712cbc99c5d] 10.0.2.15 - - [04/Dec/2015 15:41:51] "POST /v2/722380ff5288483191cd2712cbc99c5d/volumes/999a256e-aadf-41b6-88f3-1412e3462cbe/action HTTP/1.1" 403 429 0.005410

[policy.json(cinder)]
The follwong line defines that 'migrate_volume_completion' can be executed by admin only.

"volume_extension:volume_admin_actions:migrate_volume_completion": "rule:admin_api",

[Environment]
nova: commit 7df427fdb4d7a314b79b1e3977aeb86f019b5186(master)
cinder: commit 707902ca482d1653072c2cf419c566cd5fee1f96(master)
OS: Ubuntu 14.04

stack@devstack-master:/opt/devstack$ keystone user-role-list --user demo --tenant demo
+----------------------------------+-------------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+-------------+----------------------------------+----------------------------------+
| dec5e6b197d6424bbfb3854f6718cef0 | Member | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
| de75e50dec2e48b4a4c66feef8f53432 | anotherrole | 181cd8bc1c004030aec893fc7de79618 | 722380ff5288483191cd2712cbc99c5d |
+----------------------------------+-------------+----------------------------------+----------------------------------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253363

[Davanum Srinivas (DIMS) (dims-v)]

Is there anything to do in Nova?

[Matt Riedemann (mriedem)]

ML thread on this issue:

http://lists.openstack.org/pipermail/openstack-dev/2016-March/090957.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299715

[OpenStack Infra (hudson-openstack)]

Change abandoned by Takashi NATSUME (<email address hidden>) on branch: master
Review: https://review.openstack.org/253363

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f738483e843fc27379b85c5401859ccc854adc5e
Submitter: Jenkins
Branch: master

commit f738483e843fc27379b85c5401859ccc854adc5e
Author: Matt Riedemann <email address hidden>
Date: Wed Mar 30 22:07:47 2016 -0400

    Make swap-volume an admin-only API by default

    Cinder's volume migration API is, by default, an admin-only operation.
    This includes the migrate_volume_completion API.

    When Cinder is doing a volume migration, it calls Nova's swap-volume
    API to detach the old volume that we're migrating from and attach
    the volume that we're migrating to. Then Nova calls Cinder's
    migrate_volume_completion API to signal Nova is done and Cinder
    can finish the volume migration.

    The problem is that swap-volume is not an admin-only API in Nova
    per the default policy. So if a non-admin user tries to perform
    a swap-volume operation, it will fail with a 403 when calling
    Cinder's migrate_volume_completion API, since that requires an
    admin user.

    Also, because of 98739761f17b5e0b32abd8cd262f5beda030f886 we can't
    simply avoid calling migrate_volume_completion for non-migration
    cases because that API handles the actual detach/attach for the old
    and new volumes, swap-volume is broken without calling that.

    So given swap-volume relies on an admin-only Cinder API, and is called
    from an admin-only Cinder operation (volume migration), we should
    just make it default to admin-only also.

    Change-Id: Iac03258735f3d856a474ab96fe9b0a087e32906f
    Closes-Bug: #1522705

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/596489

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596489
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=65d9d4ebc28912437a2d8e7824fd7eec15848862
Submitter: Zuul
Branch: master

commit 65d9d4ebc28912437a2d8e7824fd7eec15848862
Author: Matt Riedemann <email address hidden>
Date: Fri Aug 24 17:28:03 2018 -0400

    api-ref: fix volume attachment update policy note

    Since change Iac03258735f3d856a474ab96fe9b0a087e32906f the
    PUT /servers/{server_id}/os-volume_attachments/{volume_id}
    (swap volume) API is admin-only by default. This fixes the
    documentation in the API reference.

    Change-Id: I9e7fe40536cf5f03d97eb80fd4459b5a74e92f76
    Related-Bug: #1522705