Image location information is restricted to Admin only, causing slow snapshots with RBD

[Xav Paice]

Charm 18.05, Queens on Xenial with Ceph and kvm virt.

When using a user account with only the Member role, Snapshots of instances based on Glance images take ~15-20 mins to complete. As a user with Admin, we could not reproduce.

As a non-admin user, an 'openstack image show XXX' shows properties as "locations='[]'" . As a user with the Admin role, we see "direct_url='rbd://uuid-redacted/glance/uuid-redacted/snap', locations='[{u'url': u'rbd://uuid-redacted/glance/uuid-redacted/snap', u'metadata': {}}]'

More details in https://pastebin.canonical.com/p/cVm5dXzYZW/

This behavior is confirmed as by design in policy.json which has the entry:
"get_image_location": "role:admin",

Unfortunately, if the location isn't visible, Nova is unable to read the parent image info and use that information to determine how to make a snapshot of an instance, so defaults to the non-rbd method of downloading the image and uploading the snapshot.

[Hua Zhang (zhhuabj)]

The policy entry '"get_image_location": "role:admin"' is set by L565 [1], should we provide one option to configure it for some customization cases ?

[1] https://github.com/openstack/charm-glance/blob/stable/18.05/hooks/glance_utils.py#L565

[Frode Nordahl (fnordahl)]

Seems like there are conflicting concerns here, and that we need to revisit bug 1699565.

The patch for that bug indeed makes it impossible to configure access to image location to end users of a cloud and in effect renders the pre-existing configuration option `expose-image-locations` non-effective.

There are known security trade-offs in exposing these locations, but it is also currently the only way to take advantage of copy-on-write cloning of glance images with Ceph backed block storage in OpenStack.

[Hua Zhang (zhhuabj)]

@Xav,

The code [1] shows nova has been using admin credentials to lookup image location, what's the value of your show_multiple_locations [2] now ?

[1] https://github.com/openstack/nova/blob/stable/queens/nova/compute/manager.py#L3322
[2] https://github.com/openstack/glance/blob/stable/queens/glance/api/v2/images.py#L891

[Xav Paice (xavpaice)]

in glance-api.conf, show_multiple_locations = True

[Edward Hope-Morley (hopem)]

I've had a look into this for myself and have a few thoughts. Firstly, this issue is not constrained to snapshots it affects anything that Nova does that requires information about image locations e.g. cloning. Currently our restriction to admin only means that all non-admin users are unable to create COW cloned root disks when booting VMs which means that every boot will result in an initial download and convert to raw then upload which takes ages (and thats's per compute). To make matters worse, nova-compute will only ever attempt a clone once and fall back to using the cache if the image exists locally. So even if you fix this problem you will need to ensure that the image cache (/var/lib/nova/instances/_base) does not contain a local copy of the image you were hoping clone.

I can understand that some operators will want to restrict what non-admin users can see but the way things stand the charm is breaking nova-compute for all non-admin users that want to leverage this feature. In my opinion a better option for now would be to allow to charm to optionally set get_image_locations to admin-only so that everyone gets what they want. On the Nova side, clearly since this is still the default upstream Nova does not have support to perform admin operations on Glance (since it has never had to) so we should also look at introducing that support as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/598985

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/598985
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=9e2af26b6d10c145dcb51dfb9e54bd8b2906bcfa
Submitter: Zuul
Branch: master

commit 9e2af26b6d10c145dcb51dfb9e54bd8b2906bcfa
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 31 15:25:27 2018 +0100

    Make image location ops policy configurable

    Some users may not want *_image_location operations to be
    restricted to role:admin so this patch allows that to be
    configurable and sets the default to be False since
    enabling this by default is breaking RBD COW clones in
    Nova for non-admin users (and anywhere else that relies
    on that information).

    Change-Id: I8c293d6036bc1d6104dab5458f6915968459a09e
    Closes-Bug: #1786144

[Nobuto Murata (nobuto)]

> {'get_image_location': 'role:admin'}

It may be an awkward question, but isn't is straightforward if we allow the operation to admin plus nova and cinder user under "service"domain?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.05
Review: https://review.openstack.org/599964

[Edward Hope-Morley (hopem)]

@nobuto n-c does not have service_credentials so can only use the token from the caller.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/18.05
Review: https://review.openstack.org/599964
Reason: This is now included in the 18.08 release. If there is still a need for this to be backported to 18.05 then I can re-open this patch.