Assisted snapshots API usage broken after nova client update

Bug #1691362 reported by Silvan Kaiser on 2017-05-17
This bug affects 1 person
Affects Status Importance Assigned to Milestone
György Szombathelyi

Bug Description

Cinder change updated the Nova client in Cinder. However this broke authentication of the assisted snapshot API usage in a few drivers (NFS, Quobyte, probably more). Log example at :

2017-05-07 20:00:40.876 3080 DEBUG nova.osapi_compute.wsgi.server [req-f370f8a3-f32e-49f4-ac30-69a79604be0d - -] (3080) accepted ('', 59626) server /usr/local/lib/python2.7/dist-packages/eventlet/
2017-05-07 20:00:40.951 3080 DEBUG nova.api.openstack.wsgi [req-82410317-b9ce-4049-ba59-b58f0ca3f5fc tempest-VolumesSnapshotTestJSON-1285119241 tempest-VolumesSnapshotTestJSON-1285119241] Action: 'create', calling method: <bound method AssistedVolumeSnapshotsController.create of <nova.api.openstack.compute.assisted_volume_snapshots.AssistedVolumeSnapshotsController object at 0x7ffb4f2cb410>>, body: {"snapshot": {"create_info": {"snapshot_id": "2d10862b-1719-4357-9e98-f2906f7c9958", "type": "qcow2", "new_file": "volume-cdac1fb9-6e97-4dec-a818-dca14046933e.2d10862b-1719-4357-9e98-f2906f7c9958"}, "volume_id": "cdac1fb9-6e97-4dec-a818-dca14046933e"}} _process_stack /opt/stack/nova/nova/api/openstack/
2017-05-07 20:00:40.954 3080 DEBUG nova.policy [req-82410317-b9ce-4049-ba59-b58f0ca3f5fc tempest-VolumesSnapshotTestJSON-1285119241 tempest-VolumesSnapshotTestJSON-1285119241] Policy check for os_compute_api:os-assisted-volume-snapshots:create failed with credentials {'service_roles': [], 'user_id': u'336c34bdc6e84ce9a850b69688da3366', 'roles': [u'Member'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': False, 'project_id': u'e6bd0e3ffb6f4bc5a64ec1c3806fa19e', 'project_domain_id': u'default'} authorize /opt/stack/nova/nova/
2017-05-07 20:00:40.955 3080 DEBUG nova.api.openstack.wsgi [req-82410317-b9ce-4049-ba59-b58f0ca3f5fc tempest-VolumesSnapshotTestJSON-1285119241 tempest-VolumesSnapshotTestJSON-1285119241] Returning 403 to user: Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed. __call__ /opt/stack/nova/nova/api/openstack/
2017-05-07 20:00:40.964 3080 INFO nova.osapi_compute.wsgi.server [req-82410317-b9ce-4049-ba59-b58f0ca3f5fc tempest-VolumesSnapshotTestJSON-1285119241 tempest-VolumesSnapshotTestJSON-1285119241] "POST /v2.1/os-assisted-volume-snapshots HTTP/1.1" status: 403 len: 486 time: 0.0866740
2017-05-07 20:00:41.195 3080 INFO nova.osapi_compute.wsgi.server [req-57b534cb-59fe-42b4-9920-378150d14cbe tempest-TestShelveInstance-619862968 tempest-TestShelveInstance-619862968] "DELETE /v2.1/os-security-groups/f09f8262-202f-443f-b491-3b6e960254a5 HTTP/1.1" status: 202 len: 332 time: 0.4105690
2017-05-07 20:00:41.198 3079 DEBUG nova.osapi_compute.wsgi.server [req-b90f26

Silvan Kaiser (2-silvan) wrote :

Testing this with a revert of the novaclient change at:

description: updated
Lucian Petrut (petrutlucian94) wrote :

IMO the only issue is that this should've followed the standard deprecation policy, having the old config options working for another release.

Silvan Kaiser (2-silvan) wrote :

The change does deprecate the old options and should still work with those. So i think that part is correct.
However I tested with different configurations, including the new [nova] section settings, but this did not work. So option depreciation or not, there's either an issue with the change or a different configuration required that i failed to see.
If it's a configuration issue i'm happy to hear about it. Much better to update the configs and run with those than having to return to this change in a new one.

Silvan Kaiser (2-silvan) wrote :

Update: Things work ok for me, too, when using Lucians config and altering everything to run via admin user/project. In all other variants authentication fails. So I see two issues:
1) Backward compatibility should show a deprecation warning but not outright fail, or not?
2) I think the [nova] section should still work with the service project / priv. user settings but I might be wrong in that.

Thanks Lucian for pointing out the working variant!

György Szombathelyi (gyurco) wrote :

Well, you're right, the os_xx settings are not working with the change anymore, but do you have a cinder log with the filled in [nova] section? It should get a nova admin session correctly.

György Szombathelyi (gyurco) wrote :

Btw, can you try it with nova.auth_type=password instead of v3password?

Silvan Kaiser (2-silvan) wrote :

@György it's working with a nova section and v3password. In this case the admin project and user have to be used. I tested with the service project and the nova user (privileged) in the nova section previously which did not work.That included auth_type=password, not v3password.

György Szombathelyi (gyurco) wrote :

That's really strange, would be good to have a cinder-volume debug log to see if the authentication succeeds with the service project.

György Szombathelyi (gyurco) wrote :

Or the nova-api log, which nicely shows why the policy check failed, like in #1

Policy check for os_compute_api:os-assisted-volume-snapshots:create failed with credentials {'service_roles': [], 'user_id': u'336c34bdc6e84ce9a850b69688da3366', 'roles': [u'Member'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': False, 'project_id': u'e6bd0e3ffb6f4bc5a64ec1c3806fa19e', 'project_domain_id': u'default'} authorize /opt/stack/nova/nova/

György Szombathelyi (gyurco) wrote :

Maybe it has something to do with the service_token_roles_required = True in nova.keystone_authtoken? (Just thinking loud.)

Fix proposed to branch: master

Changed in cinder:
assignee: nobody → György Szombathelyi (gyurco)
status: New → In Progress
György Szombathelyi (gyurco) wrote :

Also a devstack patch with the new settings:

Would be good to see a CI run with an assisted snapshot capable backend.

Submitter: Jenkins
Branch: master

commit f11b51b55ec6b0a34e8dbbf441579ff33791234b
Author: Gyorgy Szombathelyi <email address hidden>
Date: Tue May 23 13:31:24 2017 +0200

    Use the deprecated os_privileged_xxx settings

    Correctly deprecate the os_privileged_xxx settings, and use them
    if still configured.

    Change-Id: Ie27f3b528dbfaa57fe354a84a93787e1618182a3
    Closes-bug: #1691362

Changed in cinder:
status: In Progress → Fix Released

This issue was fixed in the openstack/cinder development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers