volume glance metadata not included in backups.

[Dermot Tynan]

The Cinder boot-from-volume system retains the Glance metadata in the volume_glance_metadata table to ensure that the volume is correctly identified as bootable, and to retain any billing metadata forwarded by Glance. The data is copied in the case of a snapshot, so that volumes created from that snapshot also retain the glance metadata, but this is not true of volume backups.

If you back up a volume and then restore from that backup to a new volume, the Glance metadata is no longer available, and the restored volume will not be marked as bootable.

[stephen mulcahy (stephen-mulcahy)]

For normal metadata, this can be handled by storing the metadata with the backup in swift during creation, and restoring during a restore operation. The metadata can either be added to the existing backup metadata file or in a separate file. I don't think there is any good reason not to add it to the existing backup metadata file. There may be some metadata that should be excluded from this process if it doesn't apply to anything but the original volume.

For the glance metadata, the solution is not as trivial. The glance metadata, particularly the billing metadata, needs to be stored in a way that it cannot be tampered with by the user (since some of the metadata relates to licensing and so on). Volume backups are stored in user's space in swift and can thus be modified by the user, precluding us from using the same solution as for normal metadata. There are a number of potential alternatives detailed below,

1. If possible with keystone, during backup the glance metadata could be signed with a credential associated with the openstack administrator and then written with the backup in the normal way (either to the existing backup metadata file or to a separate file). On restore, the signature could be verified and if the metadata has been found to have been tampered with, either the restore fails or the glance metadata is omitted from the restore.

2. The backup process could be modified to backup the glance metadata to a separate area of swift, inaccessible to all but administrative users. During the restore, the data could be re-read from there.

3. During a restore, the glance metadata for the original volume could be re-read from the database for the original volume and copied/associated with the new volume. This obviously has the limitation that the glance metadata is not "backed up" outside of the volumes database and will thus be lost in the event of a catastrophic failure of the cluster (which the redundant backup on swift would hopefully survive). Is that a problem or is the glance metadata inherently tied to a single openstack system anyway?

Approaches 1. and 2. may also be insufficient to handle a restore to an entirely new openstack system unless sufficient data is backed up to allow administrator credentials to be restored/reconstructed on the new system (although again 1 has the benefit that digital signature verification could be performed with a copy of the administrator's public key, which could be stored with each backup).

[Malini Bhandaru (malini-k-bhandaru)]

Interesting use case for encryption and key manager.

Alternative 1 is the best, it supports a complete, self contained backup, with tamper-proof evidence.
The proposed Key manager in openstack could be used to hold all the public certificates for the various nodes in OpenStack.
But it could also be used to store a symmetric private key associated with the backup process which could be used to encrypt the meta-data and prevent tamper. The advantage of using the symmetric key is that should you clone the backup agent for high availability, they could still all use the same symmetric key. Should at some point we re-key, the original key would be "de-activated" and used only for decryption purposes (key manager API support for the same).

[Avishay Traeger (avishay-il)]

I am out of the office until 05/12/2013.

For technical issues regarding the Storwize/SVC Cinder driver, please
contact: Jie Ping Wu <email address hidden>, Li Min Liu <email address hidden>,
Ronen Kat <email address hidden>
For all other issue, please contact my manager, Dalit Naor
<email address hidden>

Note: This is an automated response to your message "[Bug 1137908] Re:
volume glance metadata not included in backups." sent on 23/04/2013
11:54:33.

This is the only notification you will receive while this person is away.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/51900

[Edward Hope-Morley (hopem)]

Created the following blueprint for the work required here:

https://blueprints.launchpad.net/cinder/+spec/cinder-backup-volume-metadata-support

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51900
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=432a47cbc03414ca7ee6565c9dbcc91d7bf675fb
Submitter: Jenkins
Branch: master

commit 432a47cbc03414ca7ee6565c9dbcc91d7bf675fb
Author: Edward Hope-Morley <email address hidden>
Date: Tue Oct 15 19:01:06 2013 +0100

    Add support for backing up volume metadata

    This commit adds a new versioned api to the backup driver base
    class which all backup driver implementations should use in order
    to support backing up volume metadata as well as data. The general
    rule here is that we backup all metadata from a set of db tables
    and can then be selective about what we restore.

    As part of this commit, Ceph backup driver implements the new api
    to backup volume metadata (including glance metadata) thus
    supporting restoring volumes as bootable.

    It is expected that the other backup driver implementations will
    follow suit and adopt this approach.

    Implements: blueprint cinder-backup-volume-metadata-support
    Fixes: bug 1137908
    Change-Id: Icf2f1202d827d2435e09f5dad0f6f022f35dceee