Here's a draft OSSN: Best practices when configuring Glance with COW backends --- ### Summary ### When deploying Glance in a popular configuration where Glance shares a common storage backend with Nova and/or Cinder, it is possible to open some known attack vectors by which malicious data modification can occur. This note reviews the known issues and suggests a Glance deployment configuration that can mitigate such attacks. ### Affected Services / Software ### Glance, all releases Nova, all releases Cinder, all releases ### Discussion ### This note applies to you if you are operating an end-user-facing glance-api service with the 'show_multiple_locations' option set to True (the default value is False) or if your end-user-facing glance-api has the 'show_image_direct_url' option set to True (default value is False). Your exposure is less if you have *only* 'show_image_direct_url=True', but the deployment configuration suggested below is recommended for your case as well. The attack vector was originally outlined in OSSN-0065 [0], though that note was not clear about the attack surface or mitigation, and contained some forward-looking statements that were not fulfilled. (Though it does contain a useful discussion of image visibility and its associated policy settings.) The subject of OSSN-0065 is "Users of Glance may be able to replace active image data", but it suggests that this is only an issue when users do not checksum their image data. It neglects the fact that in some popular deployment configurations in which Nova creates a root disk snapshot, data is never uploaded to Glance, but instead a snapshot is created directly in the backend and Nova creates a Glance image record with size 0 and no os_hash_value [1], making it impossible to compare the hash of downloaded image data to the value maintained by Glance. Further, when Nova is so configured, Nova efficiently creates a root disk directly in the backend without checksumming the image data (which is not necessarily a flaw, it's the whole point of this configuration). Similarly, when using a shared backend, or a cinder glance_store, Cinder will efficiently clone a volume created from an image directly in the backend without checksumming the image data. The attack vector is the one outlined by OSSN-0065, namely: [A] malicious user could create an image in Glance, set an additional location on that image pointing to an altered image, then delete the original location, so that consumers of the original image would unwittingly be using the malicious image. Note, however, that this attack vector cannot change the original image's checksum, and it is limited to images that are owned by the attacker. OSSN-0065 suggested that this attack vector could be addressed by using policies, but that turned out not to be the case. The only way currently to close this vector is to deploy an internal-only-facing glance-api used by Nova and Cinder, with show_multiple_locations enabled, and an end-user-facing glance-api with show_multiple_locations disabled. This was suggested in "Known Issues" in Glance release notes in the Rocky [2] through Ussuri releases, but it seems that they have not received sufficient attention. Hence this security note. So far the focus has been on show_multiple_locations. When that setting is disabled in Glance, it is not possible to manipulate the locations via the OpenStack Images API. It is worth mentioning, however, that enabling 'show_image_direct_url' (which can be used by various OpenStack services to consume images directly from the storage backend) leaks information about the backend to end users, which is never a good thing from a security point of view. We therefore recommend that OpenStack services that require exposure of the 'direct_url' image property be similarly configured to use an internal-only-facing glance-api. (End users who wish to download an image do not require access to the direct_url image property because they can simply use the image data download API call [3].) ### Recommended Actions ### A glance-api service with 'show_multiple_locations' enabled should *never* be exposed directly to end users. This setting should only be enabled on an internal-only-facing glance-api that is used by OpenStack services that require access to image locations. Similarly, enabling 'show_image_direct_url' exposes information about the storage backend that could be of use to malicious actors in as yet unknown exploits, so it should likewise only be enabled on an internal-only-facing glance-api. As this addresses a known issue, it is not an embargoed note concerning a zero-day exploit. If, however, you are learning about this for the first time, and you are exposing image locations to end users, it is possible to limit the scope of the exploit described herein immediately by restricting Glance policies related to image sharing: - "publicize_image" governs the ability to make an image available to all users in a cloud, and such images appear in the default image-list response for all users. It is restricted by default to be admin-only. - "communitize_image" governs the ability to make an image available to all users, though it does not appear in the default image-list response for all users. The default configuration allows any image owner to do this. - "add_member" governs the ability to share an image with particular other projects. The default configuration allows any image owner to do this. Restricting these to admin-only would limit the exploit to a single project, but given that it still allows for a disgruntled user to maliciously modify images within that project, it is not recommended as a long term solution. ### Notes / References ### [0] OSSN-0065: https://wiki.openstack.org/wiki/OSSN/OSSN-0065 [1] The Glance "multihash" metadata pair of 'os_hash_algo' and 'os_hash_value' were introduced in Rocky to replace the legacy md5 'checksum' field. The python-glanceclient has used multihash checksumming for download verification since version 2.13.0. [2] https://docs.openstack.org/releasenotes/glance/rocky.html#known-issues [3] https://docs.openstack.org/api-ref/image/v2/index.html?#download-binary-image-data ### Contacts / References ### Author: Brian Rosmaita, Red Hat This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0090 Original LaunchPad Bug : https://bugs.launchpad.net/ossn/+bug/1990157 Mailing List : [Security] tag on