remotefs fails to make Nova-assisted snapshot

[Evgeny Antyshev]

Tempest tests creating snapshots fail in Vzstorage CI
sample run and configuration: http://openstack-3rd-party-storage-ci-logs.virtuozzo.com/58/430858/5/check/dsvm-tempest-kvm/d959ccb

May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: DEBUG novaclient.v2.client [req-916f4da9-661e-441c-8efd-782d410cd1ce tempest-VolumesSnapshotTestJSON-2088544688 None] RESP: [403] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 131 X-Compute-Request-Id: req-afc0744f-7721-4ded-b8fa-4eedcae8d10b Date: Fri, 12 May 2017 08:38:05 GMT Connection: keep-alive
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: RESP BODY: {"forbidden": {"message": "Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed.", "code": 403}}
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: {{(pid=57716) _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:395}}
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: DEBUG novaclient.v2.client [req-916f4da9-661e-441c-8efd-782d410cd1ce tempest-VolumesSnapshotTestJSON-2088544688 None] POST call to compute for http://10.161.193.63:8774/v2.1/os-assisted-volume-snapshots used request id req-afc0744f-7721-4ded-b8fa-4eedcae8d10b {{(pid=57716) request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:640}}
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs [req-916f4da9-661e-441c-8efd-782d410cd1ce tempest-VolumesSnapshotTestJSON-2088544688 None] Call to Nova to create snapshot failed
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs Traceback (most recent call last):
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/opt/stack/new/cinder/cinder/volume/drivers/remotefs.py", line 1374, in _create_snapshot_online
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs connection_info)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/opt/stack/new/cinder/cinder/compute/nova.py", line 168, in create_volume_snapshot
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs create_info=create_info)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/usr/lib/python2.7/site-packages/novaclient/v2/assisted_volume_snapshots.py", line 43, in create
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs return self._create('/os-assisted-volume-snapshots', body, 'snapshot')
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs resp, body = self.api.client.post(url, body=body)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 229, in post
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs return self.request(url, 'POST', **kwargs)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 80, in request
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs raise exceptions.from_response(resp, body, url, method)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs Forbidden: Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed. (HTTP 403) (Request-ID: req-afc0744f-7721-4ded-b8fa-4eedcae8d10b)
May 12 11:38:05 host-10-161-193-63 cinder-volume[57523]: ERROR cinder.volume.drivers.remotefs

Nova-api:
May 12 11:38:05 host-10-161-193-63 nova-api[50095]: DEBUG nova.policy [req-afc0744f-7721-4ded-b8fa-4eedcae8d10b tempest-VolumesSnapshotTestJSON-2088544688 tempest-VolumesSnapshotTestJSON-2088544688] Policy check for os_compute_api:os-assisted-volume-snapshots:create failed with credentials {'service_roles': [], 'user_id': u'228fd60b54e54f959d36fab497920e50', 'roles': [u'Member'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': False, 'project_id': u'a6b1b0e006e3413990d380db6fb74cd7', 'project_domain_id': u'default'} {{(pid=50256) authorize /opt/stack/new/nova/nova/policy.py:168}}
May 12 11:38:05 host-10-161-193-63 nova-api[50095]: DEBUG nova.api.openstack.wsgi [req-afc0744f-7721-4ded-b8fa-4eedcae8d10b tempest-VolumesSnapshotTestJSON-2088544688 tempest-VolumesSnapshotTestJSON-2088544688] Returning 403 to user: Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed. {{(pid=50256) __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1041}}
May 12 11:38:05 host-10-161-193-63 nova-api[50095]: INFO nova.osapi_compute.wsgi.server [req-afc0744f-7721-4ded-b8fa-4eedcae8d10b tempest-VolumesSnapshotTestJSON-2088544688 tempest-VolumesSnapshotTestJSON-2088544688] 10.161.193.63 "POST /v2.1/os-assisted-volume-snapshots HTTP/1.1" status: 403 len: 486 time: 0.0821240

[Maxim Nestratov (mnestratov)]

Looks like a duplicate of https://bugs.launchpad.net/cinder/+bug/1691362

[Maxim Nestratov (mnestratov)]

Well, not. It's simply related. By default nova, for some reason, allows this actions only for admin user. Look here:

nova/policies/assisted_volume_snapshots.py

POLICY_ROOT = 'os_compute_api:os-assisted-volume-snapshots:%s'

assisted_volume_snapshots_policies = [
    policy.RuleDefault(
        name=POLICY_ROOT % 'create',
        check_str=base.RULE_ADMIN_API),
    policy.RuleDefault(
        name=POLICY_ROOT % 'delete',
        check_str=base.RULE_ADMIN_API),
    policy.RuleDefault(
        name=POLICY_ROOT % 'discoverable',
        check_str=base.RULE_ANY),
]

Thus, if you really want other users to perform this action you should allow it in your customized policy file.

[Attila Fazekas (afazekas)]

I guess the `some reason` the API is expected to called by only a service user on cider side. cinder supposed to decided about is the user allowed to do snapshot or not, than call nova `as admin` in case he is allowed.

[Matt Riedemann (mriedem)]

Hmm, I'm pretty sure the NFS CI job is creating snapshots using this API, but the difference is probably that cinder is configured to call nova with a user that has the admin role. Is the VZ CI setup differently?

[Uemit Seren (timeu)]

Running into the same issue when running tempest tests. This is with OSP12 (Pike)

[melanie witt (melwitt)]

Based on the previous comments, I don't think this is a nova bug, but rather a configuration issue. I think in order to use the assisted snapshots, you need to have configured the nova user in the cinder.conf, similar to how it's done in the legacy-tempest-dsvm-full-devstack-plugin-nfs job [1] (change this to match your environment):

[nova]
region_name = RegionOne
memcached_servers = localhost:11211
signing_dir = /var/cache/cinder
cafile = /opt/stack/data/ca-bundle.pem
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secretservice
username = nova
auth_url = http://104.239.173.182/identity
auth_type = password

Can you try that and let us know if it solves your issue?

[1] http://logs.openstack.org/14/608914/1/check/legacy-tempest-dsvm-full-devstack-plugin-nfs/7fd25dc/logs/etc/cinder/cinder.conf.txt.gz

[Uemit Seren (timeu)]

Tested it with the above [nova] settings in cinder.conf and now the tempest test finishes successfully. So it seems to be a cinder configuration issue

[Aleks (iffster)]

Same issue here on Victoria and [nova] part in cinder.conf fixed it