ScaleIO volumes contain previous data

Bug #1699573 reported by Martin Chlumsky
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
tssgery
OpenStack Security Advisory
Won't Fix
Undecided
Unassigned
OpenStack Security Notes
Fix Released
Undecided
Unassigned

Bug Description

ScaleIO driver does not clear the volume after deletion when the following configuration is set in cinder.conf:

[DEFAULT]

(...)

# Method used to wipe old volumes (string value)
# Allowed values: none, zero, shred
volume_clear=zero

# Size in MiB to wipe at start of old volumes. 1024 MiBat max. 0 => all
# (integer value)
# Maximum value: 1024
volume_clear_size=8

Asking on IRC, it appears this feature is not implemented in the ScaleIO driver.

Would it be possible to implement it?

We have the Zero padding feature disabled because of concerns over performance and we are getting newly created volumes that have pre-existing filesystems on them.

With this feature, we could quickly wipe the beginning of the volume and the filesystem would be gone.

CVE References

Revision history for this message
Eric Harney (eharney) wrote :

The volume_clear option is not appropriate for scaleio, it's only for drivers where the data path is managed by cinder volume (LVM, block device driver, etc.)

If scaleio is provisioning volumes that have pre-existing data on them, that is a serious bug in the scaleio backend or driver.

It should not expose data of previously deleted volumes, and whatever is needed to fix that behavior should not be optional.

summary: - ScaleIO: Add volume_clear functionality
+ ScaleIO volumes contain previous data
information type: Public → Public Security
Revision history for this message
Eric Harney (eharney) wrote :

If true, this is a security issue, marking as such.

Changed in cinder:
importance: Undecided → High
Changed in ossa:
status: New → Incomplete
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Is this something that can be fixed/mitigated by Cinder?

Revision history for this message
Eric Harney (eharney) wrote :

Not sure, we need input from the ScaleIO driver owners.

Revision history for this message
Xing Yang (xing-yang) wrote :

Assigned to Eric Young. Eric Y., please take a look. Thanks.

Changed in cinder:
assignee: nobody → tssgery (eric-aceshome)
Xing Yang (xing-yang)
Changed in cinder:
status: New → Triaged
Revision history for this message
tssgery (eric-aceshome) wrote :

This can occur with thick volumes when zero padding is disabled in the storage pool, it can be worked around within the scaleio volume driver and that enhancement isplanned for the Queens release.

If thin volumes are used, this does not occur.

tssgery (eric-aceshome)
Changed in cinder:
status: Triaged → In Progress
Revision history for this message
Jeremy Stanley (fungi) wrote :

Is the patch to correct this behavior one which can be backported safely to prior releases/stable branches? If not, and the only mitigation is to adjust configuration, then this information is probably better suited as a security note than an advisory.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/494702

Revision history for this message
Jeremy Stanley (fungi) wrote :

Given that the proposed fix seems like it's taking a non-backportable route, I'm going to mark the security advisory task as won't fix citing report class B1 or maybe C2 per the OpenStack VMT's report taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy (which class it is depends on whether you view the cinder patch as a vulnerability fix or merely a workaround for a vulnerability in the vendor's device).

Changed in ossa:
status: Incomplete → Won't Fix
Revision history for this message
Jeremy Stanley (fungi) wrote :

I've also added a new OSSN task in case the security notes editors are interested in documenting the situation/solution once it reaches some conclusion.

Revision history for this message
tssgery (eric-aceshome) wrote :

To be clear, this is not a vulnerability in the cinder volume driver but is a side effect of having "zero padding" disabled in the storage pool within the ScaleIO system.

I am currently evaluating the possibilities of providing a workaround within the driver, including the viability of backporting.

For customers concerned about this issue, it is advised to either utilize thin volumes or enable zero padding within the ScaleIO Storage Pools.

Revision history for this message
tssgery (eric-aceshome) wrote :

This issue cannot be successfully worked around in the Cinder driver.

Users who are worried about this occurring should ensure that zero padding is enabled within the ScaleIO Storage Pool(s). Full information can be found within the Dell EMC ScaleIO documentation.

The setting can be queried by running:

scli --query_all

The command to enable zero padding is:

scli --modify_zero_padding_policy
              (((--protection_domain_id <ID> |
              --protection_domain_name <NAME>)
              --storage_pool_name <NAME>) | --storage_pool_id <ID>)
              (--enable_zero_padding | --disable_zero_padding)

Changed in cinder:
assignee: tssgery (eric-aceshome) → nobody
status: In Progress → Opinion
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Can't this be fixed in the driver by looking at the volume_clear flag and performing some of this automatically?

Revision history for this message
tssgery (eric-aceshome) wrote :

I don't believe the driver should be modifying the configuration of the ScaleIO system. These steps need to be performed by an administrator on one of the ScaleIO systems as there is no remote capability to enable this.

I do plan to check the status of zero padding and warn the users that it should be enabled. A review is already submitted that contains this warning, for the Queens release. Older releases are soon to follow.

tssgery (eric-aceshome)
Changed in cinder:
assignee: nobody → tssgery (eric-aceshome)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/504288

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Eric Young (<email address hidden>) on branch: master
Review: https://review.openstack.org/494702
Reason: This cannot be worked around in the driver. The solution is to make sure that zero padding is enabled within the ScaleIO storage pool

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/pike)

Change abandoned by Eric Young (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/504288

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.openstack.org/502473
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=aa8b87a83cc5a8cfcaa3080c9a6080d8289716a4
Submitter: Jenkins
Branch: master

commit aa8b87a83cc5a8cfcaa3080c9a6080d8289716a4
Author: Eric Young <email address hidden>
Date: Mon Sep 11 09:46:11 2017 -0400

    ScaleIO Driver - adding cache and refactoring tests

    Changing static lists to a simple cache.
    Refactoring some of the unit tests to simplify maintenance.

    Related-Bug: #1699573

    Change-Id: Idff127801da9e286a6b634594e5577eeb9782571

Revision history for this message
Nick Tait (nickthetait) wrote :

So a fix/workaround for this problem has been merged right? OK if I draft a OSSN to explain it?

Revision history for this message
Eric Harney (eharney) wrote :

A fix for this has not been landed in Cinder.

Revision history for this message
Eric Harney (eharney) wrote :

We should add a fix to the Cinder driver that refuses volume creation if this configuration is detected, to prevent leaking data between tenants. A warning message is not adequate.

Changed in cinder:
status: Opinion → New
Revision history for this message
tssgery (eric-aceshome) wrote :

There is not going to be a fix made to the cinder driver for this. The workaround is documented starting at comment #11 (utilize thin volumes or ensure zero-padding is enabled int he storage pool).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/555421

Changed in cinder:
assignee: tssgery (eric-aceshome) → Eric Harney (eharney)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/555546

Changed in cinder:
assignee: Eric Harney (eharney) → tssgery (eric-aceshome)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Eric Harney (<email address hidden>) on branch: master
Review: https://review.openstack.org/555421
Reason: https://review.openstack.org/#/c/555546/ is looking good, let's go that route. Thanks Eric

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/555546
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7feb62197d371ab7253dc86a34af6ff8b484b4df
Submitter: Zuul
Branch: master

commit 7feb62197d371ab7253dc86a34af6ff8b484b4df
Author: Eric Young <email address hidden>
Date: Thu Mar 22 20:24:01 2018 -0400

    ScaleIO: Prevent usage of unsafe volumes

    It is possible for thick volumes, created from storage pools
    which have zero-padding disabled, to contain previous data. This
    change prevents these volumes from being created by default. A
    user can override this behavior by acknowleding the possibility
    with a configuration option.

    Change-Id: I62f8f48b1624fc9abb7427bd4ca51f7873d35b96
    Closes-bug: #1699573

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 13.0.0.0b1

This issue was fixed in the openstack/cinder 13.0.0.0b1 development milestone.

Luke Hinds (lhinds)
Changed in ossn:
status: New → Fix Released
Revision history for this message
Summer Long (slong-g) wrote :

This has been assigned CVE-2017-15139.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/596879

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/queens)

Reviewed: https://review.openstack.org/596879
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f0cef07bef5ea8ed29179ee3774df5f4a634ba86
Submitter: Zuul
Branch: stable/queens

commit f0cef07bef5ea8ed29179ee3774df5f4a634ba86
Author: Eric Young <email address hidden>
Date: Thu Mar 22 20:24:01 2018 -0400

    ScaleIO: Prevent usage of unsafe volumes

    It is possible for volumes, created from storage pools
    which have zero-padding disabled, to contain previous data. This
    change prevents these volumes from being created by default. A
    user can override this behavior by acknowleding the possibility
    with a configuration option.

    This is a squash of the four commits that led to the final state in
    rocky to not allow the creation of any type of non-zero-padded volumes
    to be created. This adds a config option that defaults to the safe
    behavior. It is backporting a new config option, and a change in default
    behavior, but it should be acceptable in this case so that the security
    vulnerability can be addressed.

    Closes-Bug: #1784871

    Change-Id: I62f8f48b1624fc9abb7427bd4ca51f7873d35b96
    Closes-bug: #1699573
    (cherry picked from commit 7feb62197d371ab7253dc86a34af6ff8b484b4df)
    (cherry picked from commit 949cc46e162e00092aa85a7be921649c8dbf2bf8)
    (cherry picked from commit 8d0dea694a366cb3797748d389ca76b7864af16f)
    (cherry picked from commit 13a6689ccb7751c9f9b5c37ce0a3f75eb7665a95)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/601681

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/604105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/601681
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=6309c097e653c5f8b40e0602950d0ef54a9efb37
Submitter: Zuul
Branch: stable/pike

commit 6309c097e653c5f8b40e0602950d0ef54a9efb37
Author: Eric Young <email address hidden>
Date: Thu Mar 22 20:24:01 2018 -0400

    ScaleIO: Prevent usage of unsafe volumes

    It is possible for volumes, created from storage pools
    which have zero-padding disabled, to contain previous data. This
    change prevents these volumes from being created by default. A
    user can override this behavior by acknowleding the possibility
    with a configuration option.

    This is a squash of the four commits that led to the final state in
    rocky to not allow the creation of any type of non-zero-padded volumes
    to be created. This adds a config option that defaults to the safe
    behavior. It is backporting a new config option, and a change in default
    behavior, but it should be acceptable in this case so that the security
    vulnerability can be addressed.

    Closes-Bug: #1784871

    Change-Id: I62f8f48b1624fc9abb7427bd4ca51f7873d35b96
    Closes-bug: #1699573
    (cherry picked from commit f0cef07bef5ea8ed29179ee3774df5f4a634ba86)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/604105
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2dc52153215bb6a37532a959c5c98239be21bb56
Submitter: Zuul
Branch: stable/ocata

commit 2dc52153215bb6a37532a959c5c98239be21bb56
Author: Eric Young <email address hidden>
Date: Thu Mar 22 20:24:01 2018 -0400

    ScaleIO: Prevent usage of unsafe volumes

    It is possible for volumes, created from storage pools
    which have zero-padding disabled, to contain previous data. This
    change prevents these volumes from being created by default. A
    user can override this behavior by acknowleding the possibility
    with a configuration option.

    This is a squash of the four commits that led to the final state in
    rocky to not allow the creation of any type of non-zero-padded volumes
    to be created. This adds a config option that defaults to the safe
    behavior. It is backporting a new config option, and a change in default
    behavior, but it should be acceptable in this case so that the security
    vulnerability can be addressed.

    Closes-Bug: #1784871

    Change-Id: I62f8f48b1624fc9abb7427bd4ca51f7873d35b96
    Closes-bug: #1699573
    (cherry picked from commit f0cef07bef5ea8ed29179ee3774df5f4a634ba86)
    (cherry picked from commit 6309c097e653c5f8b40e0602950d0ef54a9efb37)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/newton)

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/606130

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.8

This issue was fixed in the openstack/cinder 10.0.8 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/newton)

Reviewed: https://review.openstack.org/606130
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=234aab19f1677a337abd4cc37ede5dc5e455f258
Submitter: Zuul
Branch: driverfixes/newton

commit 234aab19f1677a337abd4cc37ede5dc5e455f258
Author: Eric Young <email address hidden>
Date: Thu Mar 22 20:24:01 2018 -0400

    ScaleIO: Prevent usage of unsafe volumes

    It is possible for volumes, created from storage pools
    which have zero-padding disabled, to contain previous data. This
    change prevents these volumes from being created by default. A
    user can override this behavior by acknowleding the possibility
    with a configuration option.

    This is a squash of the four commits that led to the final state in
    rocky to not allow the creation of any type of non-zero-padded volumes
    to be created. This adds a config option that defaults to the safe
    behavior. It is backporting a new config option, and a change in default
    behavior, but it should be acceptable in this case so that the security
    vulnerability can be addressed.

    Closes-Bug: #1784871

    Change-Id: I62f8f48b1624fc9abb7427bd4ca51f7873d35b96
    Closes-bug: #1699573
    (cherry picked from commit f0cef07bef5ea8ed29179ee3774df5f4a634ba86)
    (cherry picked from commit 6309c097e653c5f8b40e0602950d0ef54a9efb37)
    (cherry picked from commit 2dc52153215bb6a37532a959c5c98239be21bb56)

tags: added: in-driverfixes-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.4

This issue was fixed in the openstack/cinder 12.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/625041

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/633570

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (driverfixes/newton)

Related fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/633598

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/625041
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0e9b173381fce28fded543095360dca5198c4810
Submitter: Zuul
Branch: stable/pike

commit 0e9b173381fce28fded543095360dca5198c4810
Author: Eric Young <email address hidden>
Date: Mon Sep 11 09:46:11 2017 -0400

    ScaleIO Driver - adding cache and refactoring tests

    Changing static lists to a simple cache.
    Refactoring some of the unit tests to simplify maintenance.

    Related-Bug: #1699573

    Change-Id: Idff127801da9e286a6b634594e5577eeb9782571
    (cherry picked from commit aa8b87a83cc5a8cfcaa3080c9a6080d8289716a4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.2.0

This issue was fixed in the openstack/cinder 11.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/633570
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=034feb6d749ace24fe709c9bf1aa8e9eb87a0cff
Submitter: Zuul
Branch: stable/ocata

commit 034feb6d749ace24fe709c9bf1aa8e9eb87a0cff
Author: Eric Young <email address hidden>
Date: Mon Sep 11 09:46:11 2017 -0400

    ScaleIO Driver - adding cache and refactoring tests

    Changing static lists to a simple cache.
    Refactoring some of the unit tests to simplify maintenance.

    Related-Bug: #1699573

    Change-Id: Idff127801da9e286a6b634594e5577eeb9782571
    (cherry picked from commit aa8b87a83cc5a8cfcaa3080c9a6080d8289716a4)
    (cherry picked from commit 0e9b173381fce28fded543095360dca5198c4810)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (driverfixes/newton)

Reviewed: https://review.openstack.org/633598
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=442832947f4e4968562679ab53378aae8aca560a
Submitter: Zuul
Branch: driverfixes/newton

commit 442832947f4e4968562679ab53378aae8aca560a
Author: Eric Young <email address hidden>
Date: Mon Sep 11 09:46:11 2017 -0400

    ScaleIO Driver - adding cache and refactoring tests

    Changing static lists to a simple cache.
    Refactoring some of the unit tests to simplify maintenance.

    Related-Bug: #1699573

    Change-Id: Idff127801da9e286a6b634594e5577eeb9782571
    (cherry picked from commit aa8b87a83cc5a8cfcaa3080c9a6080d8289716a4)
    (cherry picked from commit 0e9b173381fce28fded543095360dca5198c4810)
    (cherry picked from commit 034feb6d749ace24fe709c9bf1aa8e9eb87a0cff)

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.