Best practices when configuring Glance with COW backends
---
### Summary ###
When deploying Glance in a popular configuration where Glance shares a common
storage backend with Nova and/or Cinder, it is possible to open some known
attack vectors by which malicious data modification can occur. This note
reviews the known issues and suggests a Glance deployment configuration that
can mitigate such attacks.
### Affected Services / Software ###
Glance, all releases
Nova, all releases
Cinder, all releases
### Discussion ###
This note applies to you if you are operating an end-user-facing
glance-api service with the 'show_multiple_locations' option set to True
(the default value is False) or if your end-user-facing glance-api has
the 'show_image_direct_url' option set to True (default value is False).
Your exposure is less if you have *only* 'show_image_direct_url=True',
but the deployment configuration suggested below is recommended for your
case as well.
The attack vector was originally outlined in OSSN-0065 [0], though that
note was not clear about the attack surface or mitigation, and contained
some forward-looking statements that were not fulfilled. (Though it
does contain a useful discussion of image visibility and its associated
policy settings.)
The subject of OSSN-0065 is "Users of Glance may be able to replace
active image data", but it suggests that this is only an issue when
users do not checksum their image data. It neglects the fact that in
some popular deployment configurations in which Nova creates a root disk
snapshot, data is never uploaded to Glance, but instead a snapshot is
created directly in the backend and Nova creates a Glance image record
with size 0 and no os_hash_value [1], making it impossible to compare
the hash of downloaded image data to the value maintained by Glance.
Further, when Nova is so configured, Nova efficiently creates a root
disk directly in the backend without checksumming the image data (which
is not necessarily a flaw, it's the whole point of this configuration).
Similarly, when using a shared backend, or a cinder glance_store, Cinder
will efficiently clone a volume created from an image directly in the
backend without checksumming the image data.
The attack vector is the one outlined by OSSN-0065, namely:
[A] malicious user could create an image in Glance, set an additional
location on that image pointing to an altered image, then delete the
original location, so that consumers of the original image would
unwittingly be using the malicious image. Note, however, that this
attack vector cannot change the original image's checksum, and it is
limited to images that are owned by the attacker.
OSSN-0065 suggested that this attack vector could be addressed by using
policies, but that turned out not to be the case. The only way currently
to close this vector is to deploy an internal-only-facing glance-api
used by Nova and Cinder, with show_multiple_locations enabled, and an
end-user-facing glance-api with show_multiple_locations disabled.
This was suggested in "Known Issues" in Glance release notes in the
Rocky [2] through Ussuri releases, but it seems that they have not
received sufficient attention. Hence this security note.
So far the focus has been on show_multiple_locations. When that setting
is disabled in Glance, it is not possible to manipulate the locations
via the OpenStack Images API. It is worth mentioning, however, that
enabling 'show_image_direct_url' (which can be used by various OpenStack
services to consume images directly from the storage backend) leaks
information about the backend to end users, which is never a good thing
from a security point of view. We therefore recommend that OpenStack
services that require exposure of the 'direct_url' image property
be similarly configured to use an internal-only-facing glance-api.
(End users who wish to download an image do not require access to the
direct_url image property because they can simply use the image data
download API call [3].)
### Recommended Actions ###
A glance-api service with 'show_multiple_locations' enabled should
*never* be exposed directly to end users. This setting should only
be enabled on an internal-only-facing glance-api that is used by
OpenStack services that require access to image locations.
Similarly, enabling 'show_image_direct_url' exposes information about
the storage backend that could be of use to malicious actors in as yet
unknown exploits, so it should likewise only be enabled on an
internal-only-facing glance-api.
As this addresses a known issue, it is not an embargoed note concerning
a zero-day exploit. If, however, you are learning about this for the
first time, and you are exposing image locations to end users, it is
possible to limit the scope of the exploit described herein immediately
by restricting Glance policies related to image sharing:
- "publicize_image" governs the ability to make an image available
to all users in a cloud, and such images appear in the default
image-list response for all users. It is restricted by default
to be admin-only.
- "communitize_image" governs the ability to make an image available
to all users, though it does not appear in the default image-list
response for all users. The default configuration allows any
image owner to do this.
- "add_member" governs the ability to share an image with particular
other projects. The default configuration allows any image owner
to do this.
Restricting these to admin-only would limit the exploit to a single
project, but given that it still allows for a disgruntled user to
maliciously modify images within that project, it is not recommended
as a long term solution.
Here's a draft OSSN:
Best practices when configuring Glance with COW backends
---
### Summary ###
When deploying Glance in a popular configuration where Glance shares a common
storage backend with Nova and/or Cinder, it is possible to open some known
attack vectors by which malicious data modification can occur. This note
reviews the known issues and suggests a Glance deployment configuration that
can mitigate such attacks.
### Affected Services / Software ###
Glance, all releases
Nova, all releases
Cinder, all releases
### Discussion ### locations' option set to True direct_ url' option set to True (default value is False). direct_ url=True' ,
This note applies to you if you are operating an end-user-facing
glance-api service with the 'show_multiple_
(the default value is False) or if your end-user-facing glance-api has
the 'show_image_
Your exposure is less if you have *only* 'show_image_
but the deployment configuration suggested below is recommended for your
case as well.
The attack vector was originally outlined in OSSN-0065 [0], though that
note was not clear about the attack surface or mitigation, and contained
some forward-looking statements that were not fulfilled. (Though it
does contain a useful discussion of image visibility and its associated
policy settings.)
The subject of OSSN-0065 is "Users of Glance may be able to replace
active image data", but it suggests that this is only an issue when
users do not checksum their image data. It neglects the fact that in
some popular deployment configurations in which Nova creates a root disk
snapshot, data is never uploaded to Glance, but instead a snapshot is
created directly in the backend and Nova creates a Glance image record
with size 0 and no os_hash_value [1], making it impossible to compare
the hash of downloaded image data to the value maintained by Glance.
Further, when Nova is so configured, Nova efficiently creates a root
disk directly in the backend without checksumming the image data (which
is not necessarily a flaw, it's the whole point of this configuration).
Similarly, when using a shared backend, or a cinder glance_store, Cinder
will efficiently clone a volume created from an image directly in the
backend without checksumming the image data.
The attack vector is the one outlined by OSSN-0065, namely:
[A] malicious user could create an image in Glance, set an additional
location on that image pointing to an altered image, then delete the
original location, so that consumers of the original image would
unwittingly be using the malicious image. Note, however, that this
attack vector cannot change the original image's checksum, and it is
limited to images that are owned by the attacker.
OSSN-0065 suggested that this attack vector could be addressed by using only-facing glance-api locations enabled, and an locations disabled.
policies, but that turned out not to be the case. The only way currently
to close this vector is to deploy an internal-
used by Nova and Cinder, with show_multiple_
end-user-facing glance-api with show_multiple_
This was suggested in "Known Issues" in Glance release notes in the
Rocky [2] through Ussuri releases, but it seems that they have not
received sufficient attention. Hence this security note.
So far the focus has been on show_multiple_ locations. When that setting direct_ url' (which can be used by various OpenStack only-facing glance-api.
is disabled in Glance, it is not possible to manipulate the locations
via the OpenStack Images API. It is worth mentioning, however, that
enabling 'show_image_
services to consume images directly from the storage backend) leaks
information about the backend to end users, which is never a good thing
from a security point of view. We therefore recommend that OpenStack
services that require exposure of the 'direct_url' image property
be similarly configured to use an internal-
(End users who wish to download an image do not require access to the
direct_url image property because they can simply use the image data
download API call [3].)
### Recommended Actions ### locations' enabled should only-facing glance-api that is used by
A glance-api service with 'show_multiple_
*never* be exposed directly to end users. This setting should only
be enabled on an internal-
OpenStack services that require access to image locations.
Similarly, enabling 'show_image_ direct_ url' exposes information about only-facing glance-api.
the storage backend that could be of use to malicious actors in as yet
unknown exploits, so it should likewise only be enabled on an
internal-
As this addresses a known issue, it is not an embargoed note concerning
a zero-day exploit. If, however, you are learning about this for the
first time, and you are exposing image locations to end users, it is
possible to limit the scope of the exploit described herein immediately
by restricting Glance policies related to image sharing:
- "publicize_image" governs the ability to make an image available
to all users in a cloud, and such images appear in the default
image-list response for all users. It is restricted by default
to be admin-only.
- "communitize_image" governs the ability to make an image available
to all users, though it does not appear in the default image-list
response for all users. The default configuration allows any
image owner to do this.
- "add_member" governs the ability to share an image with particular
other projects. The default configuration allows any image owner
to do this.
Restricting these to admin-only would limit the exploit to a single
project, but given that it still allows for a disgruntled user to
maliciously modify images within that project, it is not recommended
as a long term solution.
### Notes / References ### /wiki.openstack .org/wiki/ OSSN/OSSN- 0065 /docs.openstack .org/releasenot es/glance/ rocky.html# known-issues /docs.openstack .org/api- ref/image/ v2/index. html?#download- binary- image-data
[0] OSSN-0065: https:/
[1] The Glance "multihash" metadata pair of 'os_hash_algo' and 'os_hash_value'
were introduced in Rocky to replace the legacy md5 'checksum' field.
The python-glanceclient has used multihash checksumming for download
verification since version 2.13.0.
[2] https:/
[3] https:/
### Contacts / References ### /wiki.openstack .org/wiki/ OSSN/OSSN- 0090 /bugs.launchpad .net/ossn/ +bug/1990157 /launchpad. net/~openstack- ossg
Author: Brian Rosmaita, Red Hat
This OSSN : https:/
Original LaunchPad Bug : https:/
Mailing List : [Security] tag on <email address hidden>
OpenStack Security Project : https:/
CVE: none