Allow for configurable policy for wiping data when deleting volumes

[Daniel Berrange]

When deleting volumes in Nova, it will fill the entire file with zeros. This can take a long time, and may be considered unnecessary in some deployment environments. On the other hand it may also be considered insufficiently paranoid. There ought to be a configurable policy for wiping data, allowing for no wiping, zeroing, or one of the scrub command algorithms. See also this thread.

https://lists.launchpad.net/openstack/msg14333.html

[Pádraig Brady (p-draigbrady)]

I was intending to look at this at some stage.
Configuration is a good thing here to support various devices and security standards.
Some notes I have for various options:

none
  Don't bother, or leave to lower layers (see discard)
zero
  dd bs=1M if=/dev/zero
shred
  # doesn't use /dev/urandom by default as very slow and overkill
  # uses 3 passes by default
  # does not yet support DoD 5220 rules
  # does not yet support ATA secure erase
discard
  # benefits on SSDs to return blocks to pool and efficiently erase
  # some devices don's zero on discard. See /sys/block/$dev/queue/discard_zeroes_data
  # lvm supports directly since 2.02.85 (04/2011) with issue_discards=1 in /etc/lvm/lvm.conf
  # manual trim call with hdparm --trim-sector-ranges. That's a bit awkward though...
  # ... must look into adding simple support to some command (fstrim, truncate,shred?)
  # Note mkfs.* generally do a trim of the device first anyway

I'd not considered a partial overwrite of the start of the device,
as discussed in the referenced email thread. Does that provide
any functional benefit?

[Daniel Berrange (berrange)]

I don't think you want to incorporate 'discard' in this. IMHO, that is a separate logical operation, that you may choose to use /in conjunction with/ one of the data-wiping operations, not instead of.

Whether wiping the start of the device is useful depends somewhat of what the guest OS has used it for. If the guest used it for a plain filesystem, then wiping will scramble much fo the filesystem metadat, but not the real data. So it is at best "Security through obscurity". If, however, the guest had put something like LUKS encryption on the volume, then wiping the start will blow away all the decryption key sots, so this will be as secure as any other method.

[Jay Pipes (jaypipes)]

++ for making this a configurable option in Nova/Cinder

[Duncan Thomas (duncan-thomas)]

HP's block store solution does scrubbing asynchronously on the the back end, so we'd certainly like to be able to tell nova/cinder not to do it itself....

[Pádraig Brady (p-draigbrady)]

I'll add a "volume_clear" option with the default value of "zero" but can be any of:

none
zero # shred -n0 -z # or dd if faster
zero_size # shred -n0 -z -s$size
shred # shred default of 3 passes
shred_n # shred -n$n

[clayg (clay-gerrard)]

On Wed, Jul 11, 2012 at 9:57 AM, Duncan Thomas
<email address hidden> wrote:
> HP's block store solution does scrubbing asynchronously on the the back
> end, so we'd certainly like to be able to tell nova/cinder not to do it
> itself....
>
>

I think the policy would be an option for the ISCSIDrver, surly the
manager will continue to just call
self.driver.delete_volume(volume_id).

-clayg

[Duncan Thomas (duncan-thomas)]

On 11 July 2012 16:40, clayg <email address hidden> wrote:
> On Wed, Jul 11, 2012 at 9:57 AM, Duncan Thomas
> <email address hidden> wrote:
>> HP's block store solution does scrubbing asynchronously on the the back
>> end, so we'd certainly like to be able to tell nova/cinder not to do it
>> itself....
>>
>>
>
> I think the policy would be an option for the ISCSIDrver, surly the
> manager will continue to just call
> self.driver.delete_volume(volume_id).

It's highly likely tahn some none-iscsi setups might want it as well
though... even if they don't know they want it until they get it...
seems an idea for ti to be at cinder level rather than driver level,
but optional, no?

--
Duncan Thomas

[Pádraig Brady (p-draigbrady)]

Also related is bug 937694

[John Griffith (john-griffith)]

p-draigbrady... any updates on this? Haven't seen anything hit for Nova , were you planning to submit this week to make RC-1?

[Pádraig Brady (p-draigbrady)]

What's the final cutoff for this.
I've a little time freeing up over the next few days,
so I could get it done for Thursday I think.

[John Griffith (john-griffith)]

Thursday would work so long as it's tested and we're confident it works well and there's minimal risk of breaking anything else that's in place.

To clarify I think the driver is the right level to have this. Not sure what Duncan was envisioning when he stated "cinder level" but the driver level gives everybody the flexibility that's needed.

I'm actually most interested in something "faster" like an "unsecure-delete" option. The dd method implemented today is pretty effective, but for those truly paranoid implementing an option to specify "shred" might be worthwhile.

One other thing to consider, would it be better to leave this out of flags and just have an optional delete-type as an arg to the delete command? This way an end user can choose what's appropriate and what's not, and a service provider admin can use flags to enable/disable this ability for their end-users? At this point maybe we're getting out of the scope of Folsom.... thoughts?

[Pádraig Brady (p-draigbrady)]

Yes I aggree that the driver level is the place. I was considering having a clear() method that could be specialised if required.

The faster method currently is specifying a small size that would clear out partition tables/encryption keys.
Also people may specify "none" and have separate async scrubber as described in comment #4 for example

As for having the clear method specified with the delete command,
i.e. overriding the default clear method, that is further functionality
I think which is less pressing (if needed at all), and could be pused to Grizzly I think.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12507

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12521

[Duncan Thomas (duncan-thomas)]

Ignore my comment about moving it up and out of the driver - after a bit of discussion on IRC I was convinced that it is trivial enough to implement in the driver for the simple case and the complex case is, by its nature, volume specific.

[Pádraig Brady (p-draigbrady)]

volume support has been completely removed to cinder now

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/12521
Committed: http://github.com/openstack/cinder/commit/bb06ebd0f6a75a6ba55a7c022de96a91e3750d20
Submitter: Jenkins
Branch: master

commit bb06ebd0f6a75a6ba55a7c022de96a91e3750d20
Author: Pádraig Brady <email address hidden>
Date: Thu Sep 6 16:36:50 2012 +0100

    support a configurable volume wiping method

    Add a clear_volume() method that is called from delete_method().
    This is only implemented at present for the base LVM driver.
    This new clear_volume() method now supports these config vars:

      volume_clear = none, zero, shred
      volume_clear_size = size_in_MiB

    'zero' is the default method and unchanged from previously.
    'none' is used if security isn't a concern, or there is
      independent volume scrubbing.
    'shred' is for more security conscious situations where
    an overwrite count is required. Currently this defaults to 3 passes.

    size_in_MiB can be used to limit the cleared area to the first
    part of the volume, which can greatly speed up the operation,
    and can be useful with encrypted volumes for example where
    overwritting the encryption keys at the start is sufficient.

    Fixes bug: 1022511
    Change-Id: I7312678ff5d66156578501483a1b74e2a69e19a0

[clayg (clay-gerrard)]

http://zerrod.com/ylujvzgf/tjprknq.ppshtkkigamkdcodfcua