Comment 19 for bug 1990157

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote : Re: Malicious image data modification can happen when using COW

Here's a draft OSSN:

Best practices when configuring Glance with COW backends
---

### Summary ###
When deploying Glance in a popular configuration where Glance shares a common
storage backend with Nova and/or Cinder, it is possible to open some known
attack vectors by which malicious data modification can occur. This note
reviews the known issues and suggests a Glance deployment configuration that
can mitigate such attacks.

### Affected Services / Software ###
Glance, all releases
Nova, all releases
Cinder, all releases

### Discussion ###
This note applies to you if you are operating an end-user-facing
glance-api service with the 'show_multiple_locations' option set to True
(the default value is False) or if your end-user-facing glance-api has
the 'show_image_direct_url' option set to True (default value is False).
Your exposure is less if you have *only* 'show_image_direct_url=True',
but the deployment configuration suggested below is recommended for your
case as well.

The attack vector was originally outlined in OSSN-0065 [0], though that
note was not clear about the attack surface or mitigation, and contained
some forward-looking statements that were not fulfilled. (Though it
does contain a useful discussion of image visibility and its associated
policy settings.)

The subject of OSSN-0065 is "Users of Glance may be able to replace
active image data", but it suggests that this is only an issue when
users do not checksum their image data. It neglects the fact that in
some popular deployment configurations in which Nova creates a root disk
snapshot, data is never uploaded to Glance, but instead a snapshot is
created directly in the backend and Nova creates a Glance image record
with size 0 and no os_hash_value [1], making it impossible to compare
the hash of downloaded image data to the value maintained by Glance.
Further, when Nova is so configured, Nova efficiently creates a root
disk directly in the backend without checksumming the image data (which
is not necessarily a flaw, it's the whole point of this configuration).
Similarly, when using a shared backend, or a cinder glance_store, Cinder
will efficiently clone a volume created from an image directly in the
backend without checksumming the image data.

The attack vector is the one outlined by OSSN-0065, namely:

   [A] malicious user could create an image in Glance, set an additional
   location on that image pointing to an altered image, then delete the
   original location, so that consumers of the original image would
   unwittingly be using the malicious image. Note, however, that this
   attack vector cannot change the original image's checksum, and it is
   limited to images that are owned by the attacker.

OSSN-0065 suggested that this attack vector could be addressed by using
policies, but that turned out not to be the case. The only way currently
to close this vector is to deploy an internal-only-facing glance-api
used by Nova and Cinder, with show_multiple_locations enabled, and an
end-user-facing glance-api with show_multiple_locations disabled.

This was suggested in "Known Issues" in Glance release notes in the
Rocky [2] through Ussuri releases, but it seems that they have not
received sufficient attention. Hence this security note.

So far the focus has been on show_multiple_locations. When that setting
is disabled in Glance, it is not possible to manipulate the locations
via the OpenStack Images API. It is worth mentioning, however, that
enabling 'show_image_direct_url' (which can be used by various OpenStack
services to consume images directly from the storage backend) leaks
information about the backend to end users, which is never a good thing
from a security point of view. We therefore recommend that OpenStack
services that require exposure of the 'direct_url' image property
be similarly configured to use an internal-only-facing glance-api.
(End users who wish to download an image do not require access to the
direct_url image property because they can simply use the image data
download API call [3].)

### Recommended Actions ###
A glance-api service with 'show_multiple_locations' enabled should
*never* be exposed directly to end users. This setting should only
be enabled on an internal-only-facing glance-api that is used by
OpenStack services that require access to image locations.

Similarly, enabling 'show_image_direct_url' exposes information about
the storage backend that could be of use to malicious actors in as yet
unknown exploits, so it should likewise only be enabled on an
internal-only-facing glance-api.

As this addresses a known issue, it is not an embargoed note concerning
a zero-day exploit. If, however, you are learning about this for the
first time, and you are exposing image locations to end users, it is
possible to limit the scope of the exploit described herein immediately
by restricting Glance policies related to image sharing:

- "publicize_image" governs the ability to make an image available
  to all users in a cloud, and such images appear in the default
  image-list response for all users. It is restricted by default
  to be admin-only.

- "communitize_image" governs the ability to make an image available
  to all users, though it does not appear in the default image-list
  response for all users. The default configuration allows any
  image owner to do this.

- "add_member" governs the ability to share an image with particular
  other projects. The default configuration allows any image owner
  to do this.

Restricting these to admin-only would limit the exploit to a single
project, but given that it still allows for a disgruntled user to
maliciously modify images within that project, it is not recommended
as a long term solution.

### Notes / References ###
[0] OSSN-0065: https://wiki.openstack.org/wiki/OSSN/OSSN-0065
[1] The Glance "multihash" metadata pair of 'os_hash_algo' and 'os_hash_value'
    were introduced in Rocky to replace the legacy md5 'checksum' field.
    The python-glanceclient has used multihash checksumming for download
    verification since version 2.13.0.
[2] https://docs.openstack.org/releasenotes/glance/rocky.html#known-issues
[3] https://docs.openstack.org/api-ref/image/v2/index.html?#download-binary-image-data

### Contacts / References ###
Author: Brian Rosmaita, Red Hat
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0090
Original LaunchPad Bug : https://bugs.launchpad.net/ossn/+bug/1990157
Mailing List : [Security] tag on <email address hidden>
OpenStack Security Project : https://launchpad.net/~openstack-ossg
CVE: none