Use of MD5 in OpenStack Glance image signature (CVE-2015-8234)

Bug #1516031 reported by Tristan Cacqueray
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Medium
Dane Fichter
OpenStack Security Advisory
Won't Fix
Undecided
Unassigned
OpenStack Security Notes
Fix Released
High
Robert Clark

Bug Description

This have been reported by Daniel P. Berrange:
"
In the OpenStack Liberty release, the Glance project added support for image signature verification.

http://specs.openstack.org/openstack/glance-specs/specs/liberty/image-signing-and-verification-support.html

The verification code was added in the following git commit

https://github.com/openstack/glance/commit/484ef1b40b738c87adb203bba6107ddb4b04ff6e

Unfortunately the design of this signature verification method is flawed by design.

The generalized approach to creating signatures of content is to apply a hash to the content and then encrypt it in some manner. Consider that the signature is defined to use hash=sha256 and cipher=rsa we can describe the signature computation as

signature = rsa(sha256(content))

In the case of verifying a disk image, the content we care about verifying is the complete disk image file. Unfortunately, the glance specification chose *not* to compute the signature against the disk image file. Glance already had an MD5 checksum calculated for the disk image file, so they instead chose to compute the signature against the MD5 checksum instead. ie glance is running

signature = rsa(sha256(md5(disk-image-content)))

This degrades the security of the system to that of the weakest hash, which is obviously MD5 here.

The code where glance verifies the signature is in the glance/locations.py, the 'set_data' method where is does

 result = signature_utils.verify_signature(
 self.context, checksum, self.image.extra_properties)
 if result:
 LOG.info(_LI("Successfully verified signature for image %s"),
 self.image.image_id)

The 'checksum' variable is populate by the glance_store driver, but it is hardcoded to always be md5 in all current glance storage backends:

 $ git grep hashlib glance_store/_drivers/ | grep checksum
 glance_store/_drivers/filesystem.py: checksum = hashlib.md5()
 glance_store/_drivers/rbd.py: checksum = hashlib.md5()
 glance_store/_drivers/s3.py: checksum = hashlib.md5()
 glance_store/_drivers/s3.py: checksum = hashlib.md5()
 glance_store/_drivers/sheepdog.py: checksum = hashlib.md5()
 glance_store/_drivers/swift/store.py: checksum =
 hashlib.md5()
 glance_store/_drivers/vmware_datastore.py: self.checksum =
 hashlib.md5()

Since we will soon be shipping OpenStack Liberty release, we need to at least give a security notice to alert our customers to the fact that the signature verification is cryptographically weak/broken. IMHO, it quite likely deserves a CVE though

NB, this is public knowledge as I first became aware of this flawed design in comments / discussion on a public specification proposed to implement the same approach in the Nova project.

My suggested way to fix this is to simply abandon the current impl and re-do it such that it directly computes the signature against the disk image, and does not use the existing md5 checksum in any way.

Regards,
Daniel
"

Mailing list thread for Nova impl: http://lists.openstack.org/pipermail/openstack-dev/2015-November/079348.html
Nova Spec: https://review.openstack.org/#/c/188874/

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

The flaw sounds legit, however the attack surface is a bit shallow:
IIUC the image signature verifies image read from a malicious backend.
End user of Glance are not really affected since they will requests the API either an image name or UUID, which are not protected by signature mechanisms afaik.
Thus the attack scenario must involves a malicious glance backend operators.

Though since this is a flawed security mechanism, it surely warrants an CVE, but not necessarly an OSSA. What do you think ?

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Assuming this feature is activated by default, here is the impact description draft:

Title: Use of MD5 in OpenStack Glance image signature
Reporter: Daniel P. Berrange (Red Hat)
Products: Glance
Affects: =11.0.0

Description:
Daniel P. Berrange from Red Hat reported a vulnerability in Glance image signature. Glance computes cryptographic signature using MD5 hash of the image. By crafting a malicious image that produces a MD5 collision, a Glance backend operator may subvert the signature verification process, resulting in corrupted image. All Glance setups are affected.

Revision history for this message
Niall Bunting (niall-bunting) wrote :

Is this a duplicate of this one from a year ago? https://bugs.launchpad.net/glance/+bug/1389772

Revision history for this message
Niall Bunting (niall-bunting) wrote :

Oh, its not thats not about the signature.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Well it's kind of related, former bug was only about using MD5 for digest. Though this bug is about using MD5 digest for crypto signature.

summary: - Use of MD5 in OpenStack Glance image signature
+ Use of MD5 in OpenStack Glance image signature (CVE-2015-8234)
Revision history for this message
Erno Kuvaja (jokke) wrote :

This is indeed weak behavior as described. I do agree that attack vector is limited to cases where there is attack vector the the backend store, but on the other hand that's exactly what the signature verification is trying to prevent.

Changed in glance:
status: New → Triaged
Revision history for this message
Erno Kuvaja (jokke) wrote :

This is also well understood issue discussed in the summit design session in Tokyo.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this does not qualify for an OpenStack Security Advisory (OSSA), I've added an OSSN task to assess if a Security Note would work better here.

Changed in ossa:
status: Incomplete → Won't Fix
Changed in ossn:
importance: Undecided → High
assignee: nobody → Robert Clark (robert-clark)
Revision history for this message
Hao Jun Wang (shhjwang) wrote :

Hi Tristan, I am a bit confused by the last sentence of the description "All Glance setups are affected". I think this CVE only affects Liberty. Maybe I have misunderstood something. Does it mean that all Glance setups with Liberty are affected? Thank you very much.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Switched bug type from vulnerability to hardening opportunity.

information type: Public Security → Public
tags: added: security
Revision history for this message
Robert Clark (robert-clark) wrote :

Is there a proposed fix / time to fix ?

I'm writing up the OSSN at the moment and this information would be useful.

Revision history for this message
Brianna Poulos (brianna-poulos) wrote :

I have submitted a glance spec for this fix: https://review.openstack.org/#/c/252462/

I am currently working on an implementation for the spec, and ideally this fix will be included in Mitaka.

Revision history for this message
Robert Clark (robert-clark) wrote :

For your consideration : Obviously a WIP - https://review.openstack.org/#/c/254427/

Revision history for this message
Robert Clark (robert-clark) wrote :

Thanks to Daniel for the excellent initial writeup

Revision history for this message
Dane Fichter (dane-fichter) wrote :

Agreeing with everything Daniel Berrange has said. My summary of the issue is as follows:

Using MD5 to compute a digest of an image and then applying a digital signature to the digest means that the digital signature of the image is only as trustworthy as MD5. Since there have been practical attacks against MD5 generating collisions, it's not safe to use MD5 in this manner with a feature intended to provide security.

We will resolve the issue in Glance and push an implementation in Nova that does not have the vulnerability by switching over to signatures which are generated by signing the image data directly (i.e. not signing the MD5 or any other digest).

Glance spec to resolve the issue in Glance by switching over to signing the image data directly: https://review.openstack.org/#/c/252462/

Nova spec describing image signature verification before booting: https://review.openstack.org/#/c/188874/

Revision history for this message
Nathan Kinder (nkinder) wrote :

This issue has been published as OSSN-0061:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0061

Changed in ossn:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/254987
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=09a0acefc7d27b85e7145611a3852bcf0765f769
Submitter: Jenkins
Branch: master

commit 09a0acefc7d27b85e7145611a3852bcf0765f769
Author: Brianna Poulos <email address hidden>
Date: Tue Dec 8 17:24:11 2015 -0500

    Add sign-the-hash deprecation warning

    Per discussion on the mailing list [1] and the related nova specification [2]
    it has been decided that the signature should be of the image data directly,
    rather than of the glance MD5 "checksum" hash of the image data.

    This patch adds TODO statements to remove the 'sign-the-hash' approach
    after the Mitaka development cycle, as well as a deprecation warning if
    the properties exist for the 'sign-the-hash' approach.

    [1] http://bit.ly/1Q0M0C7
    [2] https://review.openstack.org/#/c/188874/19

    Partial-Bug: #1516031
    Change-Id: Ifaae6e34491b382ffffe5b4e4764764d592d1f53

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/271513
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=eab1567d48a18fa968c7b66c3641dd037da1f84e
Submitter: Jenkins
Branch: master

commit eab1567d48a18fa968c7b66c3641dd037da1f84e
Author: Brianna Poulos <email address hidden>
Date: Fri Jan 22 15:13:35 2016 -0500

    Remove unneeded glance unit test assert calls

    The glance_store library now accepts a verifier object for use with
    verifying signatures, as a result of a recently merged patch [1].
    There are some glance unit tests that do not expect this verifier
    object to be passed to the add method of glance_store, and will
    therefore fail with a new release of the glance_store library.

    This patch removes the assert lines for the tests that will fail,
    since it is not necessary for glance to test glance_store in that
    way at the glance level.

    [1] https://review.openstack.org/#/c/183110/

    Change-Id: I404fbc40dafa159a63a6dfee563f05ee80c073dc
    Partial-Bug: #1516031

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/liberty)

Reviewed: https://review.openstack.org/283708
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=4f41717204dea123fac1050956cd87dea261794e
Submitter: Jenkins
Branch: stable/liberty

commit 4f41717204dea123fac1050956cd87dea261794e
Author: Brianna Poulos <email address hidden>
Date: Fri Jan 22 15:13:35 2016 -0500

    Remove unneeded glance unit test assert calls

    The glance_store library now accepts a verifier object for use with
    verifying signatures, as a result of a recently merged patch [1].
    There are some glance unit tests that do not expect this verifier
    object to be passed to the add method of glance_store, and will
    therefore fail with a new release of the glance_store library.

    This patch removes the assert lines for the tests that will fail,
    since it is not necessary for glance to test glance_store in that
    way at the glance level.

    [1] https://review.openstack.org/#/c/183110/

    Change-Id: I404fbc40dafa159a63a6dfee563f05ee80c073dc
    Partial-Bug: #1516031
    (cherry picked from commit eab1567d48a18fa968c7b66c3641dd037da1f84e)

    -- squashed with another change to get liberty working --

    Remove glance_store specific unit tests

    TestStoreAddToBackend contains a set of tests that should belong to
    the glance_store. Recently, these tests were about to break
    glance as they mocked internal methods. Those mocks were
    removed in commit eab1567d48a18fa968c7b66c3641dd037da1f84e.

    The tests are added back to glance_store in
    change https://review.openstack.org/#/c/273786.

    Conflicts:
            glance/tests/unit/test_store_image.py

    Closes-Bug: #1548902

    Depends-On: I688629d37e9146aef33fbc9bb6fd8780521e06ab
    Change-Id: Ic67826a4d02cb30cc429a3d4abc46d6ca96f31ad
    (cherry picked from commit 1ebbfd3dc1694dc4f26e763da9eee833bb5d2545)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/255584
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=95ca43f30f5bcc8107e0615b3861daa2c77a2aec
Submitter: Jenkins
Branch: master

commit 95ca43f30f5bcc8107e0615b3861daa2c77a2aec
Author: Brianna Poulos <email address hidden>
Date: Tue Dec 8 17:24:11 2015 -0500

    Add sign-the-data signature verification

    Per discussion on the mailing list [1] and the related nova
    specification [2] it has been decided that the signature should be of
    the image data directly, rather than of the glance MD5 "checksum" hash
    of the image data.

    This patch adds the ability to verify a signature of the image data
    directly, using a verifier that is passed to the glance_store backend.
    It is dependent on a glance_store patch which updates this verifier
    object with the image data as it is creating the checksum (see
    Depends-On below).

    [1] http://bit.ly/1Q0M0C7
    [2] https://review.openstack.org/#/c/188874/19

    Depends-On: I43799e6a4a6643a23769af8d839a2beb4e0ff9bf
    Partial-Bug: #1516031
    Change-Id: If0c06b3094cecef6c8ca8a65753038b6b5a9d8fe

Changed in glance:
importance: Undecided → Medium
status: Triaged → In Progress
assignee: nobody → Dane Fichter (dane-fichter)
milestone: none → newton-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/308466
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=5ab63107b69e381f04bfa4aa9143e229ac2a9857
Submitter: Jenkins
Branch: master

commit 5ab63107b69e381f04bfa4aa9143e229ac2a9857
Author: Dane Fichter <email address hidden>
Date: Tue Apr 19 01:27:02 2016 -0400

    Remove deprecated "sign-the-hash" approach

    This change removes the "sign-the-hash" signature
    verification code in the signature_utils module and
    the ImageProxy class. This code was deprecated in
    Mitaka and scheduled for removal in Newton.

    Change-Id: I8862f6c94538dd818c7360ba287e14c1264ff20f
    Closes-Bug: #1516031

Changed in glance:
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/glance 13.0.0.0b1

This issue was fixed in the openstack/glance 13.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.