the snapshot of a volume-backed instance cannot be used to boot a new instance

[Xavier Queralt]

After the changes in the block device mappings introduced for Havana, if we try to create an snapshot of a volume-backed instance the resulting image cannot be used to boot a new instance due to conflicts with the bootindex between the block_device_mapping stored in the image properties and the current image.

The steps to reproduce are:

$ glance image-create --name f20 --disk-format qcow2 --container-format bare --min-disk 2 --is-public True --min-ram 512 --copy-from http://download.fedoraproject.org/pub/fedora/linux/releases/test/20-Alpha/Images/x86_64/Fedora-x86_64-20-Alpha-20130918-sda.qcow2

$ cinder create --image-id <uuid of the new image> --display-name f20 2

$ nova boot --boot-volume <uuid of the new volume> --flavor m1.tiny test-instance

$ nova image-create test-instance test-snap

This will create an snapshot of the volume and an image in glance with a block_device_mapping containing the snapshot_id and all the other values from the original block_device_mapping (id, connection_info, instance_uuid, ...):

| Property 'block_device_mapping' | [{"instance_uuid": "989f03dc-2736-4884-ab66-97360102d804", "virtual_name": null, "no_device": null, "connection_info": "{\"driver_volume_type\": \"iscsi\", \"serial\": \"cb6d4406-1c66-4f9a-9fd8-7e246a3b93b7\", \"data\": {\"access_mode\": \"rw\", \"target_discovered\": false, \"encrypted\": false, \"qos_spec\": null, \"device_path\": \"/dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2010-10.org.openstack:volume-cb6d4406-1c66-4f9a-9fd8-7e246a3b93b7-lun-1\", \"target_iqn\": \"iqn.2010-10.org.openstack:volume-cb6d4406-1c66-4f9a-9fd8-7e246a3b93b7\", \"target_portal\": \"192.168.122.2:3260\", \"volume_id\": \"cb6d4406-1c66-4f9a-9fd8-7e246a3b93b7\", \"target_lun\": 1, \"auth_password\": \"wh5bWkAjKv7Dy6Ptt4nY\", \"auth_username\": \"oPbN9FzbEPQ3iFpPhv5d\", \"auth_method\": \"CHAP\"}}", "created_at": "2013-10-30T13:18:57.000000", "snapshot_id": "f6a25cc2-b3af-400b-9ef9-519d28239920", "updated_at": "2013-10-30T13:19:08.000000", "device_name": "/dev/vda", "deleted": 0, "volume_size": null, "volume_id": null, "id": 3, "deleted_at": null, "delete_on_termination": false}] |

When we try latter to use this image to boot a new instance, the API won't let us because both, the device in the image bdm and the image (which is empty) are considered to be the boot device:

$ nova boot --image test-snap --flavor m1.nano test-instance2
ERROR: Block Device Mapping is Invalid: Boot sequence for the instance and image/block device mapping combination is not valid. (HTTP 400) (Request-ID: req-3e502a29-9cd3-4c0c-8ddc-a28d315d21ea)

If we check the internal flow we can see that nova considers the image to be the boot device even thought the image itself doesn't define any local disk but only a block_device_mapping pointing to the snapshot.

To be able to generate proper images from volume-backed instances we should:
 1. copy only the relevant keys from the original block_device_mapping to prevent duplicities in DB
 2. prevent nova from adding a new block device for the image if this one doesn't define any local disk

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/54633

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/54634

[Xavier Queralt (xqueralt-deactivatedaccount)]

Raising importance because instance volume-backed snapshots are the only type of snapshots the EC2 API use.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54633
Committed: http://github.com/openstack/nova/commit/eddd8a14cd2393ce7bdb0ec1c7366177371192b2
Submitter: Jenkins
Branch: master

commit eddd8a14cd2393ce7bdb0ec1c7366177371192b2
Author: Xavier Queralt <email address hidden>
Date: Wed Oct 30 20:21:03 2013 +0100

    Clean BDM when snapshoting volume-backed instances

    When snapshoting a volume-backed instance, its block device mapping is
    copied entirely into the image properties of the resulting snapshot.

    This may lead to some errors when booting from that snapshot because the
    fields 'id', 'insance_uuid' and 'connection_info' must be generated for
    each instance instead of reusing the ones from the original instance.
    For example, sharing the connection_info will cause the new instance to
    try to connect to the original volume instead of creating a new one.

    This patch cleans all the database-specific plus the connection_info
    fields from the block device mapping when creating the snapshot.

    Change-Id: I4190e9519d641c61b00b5d86c3885f2ae4fa86f3
    Partial-Bug: #1246327

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54634
Committed: http://github.com/openstack/nova/commit/482dfeb113f1e0ff814aa1fc980d67e0c8b06d76
Submitter: Jenkins
Branch: master

commit 482dfeb113f1e0ff814aa1fc980d67e0c8b06d76
Author: Xavier Queralt <email address hidden>
Date: Wed Oct 30 21:25:55 2013 +0100

    Process image BDM earlier to avoid duplicates

    If the block device mapping from the image properties are not handled at
    the same time as the ones defined from the API, we might make the false
    assumption that the instance has no root and failing or, in the case
    we're booting from an image that only defines a BDM and has no disk,
    creating a local disk BDM as root and ending up with two root devices
    which is forbidden.

    This patch moves the handling of the image block device mappings to the
    same method were we check the ones provided throught the API. This
    allows nova to decide wether the instance needs a local disk from an
    image if no root device is defined in any of the block device mappings.

    Change-Id: Ide95357895ab4dd1338ab5ee3ec25294af1d010b
    Closes-Bug: #1246327

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/57890

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/57891

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57890
Committed: http://github.com/openstack/nova/commit/cad49c3f2aac14b9934ee160a650d4a7b9809d66
Submitter: Jenkins
Branch: stable/havana

commit cad49c3f2aac14b9934ee160a650d4a7b9809d66
Author: Xavier Queralt <email address hidden>
Date: Wed Oct 30 20:21:03 2013 +0100

    Clean BDM when snapshoting volume-backed instances

    When snapshoting a volume-backed instance, its block device mapping is
    copied entirely into the image properties of the resulting snapshot.

    This may lead to some errors when booting from that snapshot because the
    fields 'id', 'insance_uuid' and 'connection_info' must be generated for
    each instance instead of reusing the ones from the original instance.
    For example, sharing the connection_info will cause the new instance to
    try to connect to the original volume instead of creating a new one.

    This patch cleans all the database-specific plus the connection_info
    fields from the block device mapping when creating the snapshot.

    Change-Id: I4190e9519d641c61b00b5d86c3885f2ae4fa86f3
    Partial-Bug: #1246327
    (cherry picked from commit eddd8a14cd2393ce7bdb0ec1c7366177371192b2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57891
Committed: http://github.com/openstack/nova/commit/1d2a0b8cc640efc858242f070f96449f679b343a
Submitter: Jenkins
Branch: stable/havana

commit 1d2a0b8cc640efc858242f070f96449f679b343a
Author: Xavier Queralt <email address hidden>
Date: Wed Oct 30 21:25:55 2013 +0100

    Process image BDM earlier to avoid duplicates

    If the block device mapping from the image properties are not handled at
    the same time as the ones defined from the API, we might make the false
    assumption that the instance has no root and failing or, in the case
    we're booting from an image that only defines a BDM and has no disk,
    creating a local disk BDM as root and ending up with two root devices
    which is forbidden.

    This patch moves the handling of the image block device mappings to the
    same method were we check the ones provided throught the API. This
    allows nova to decide wether the instance needs a local disk from an
    image if no root device is defined in any of the block device mappings.

    Change-Id: Ide95357895ab4dd1338ab5ee3ec25294af1d010b
    Closes-Bug: #1246327
    (cherry picked from commit 482dfeb113f1e0ff814aa1fc980d67e0c8b06d76)

[LnxSlck (brunomiguelqueiros)]

This bug still affects me and i'm running openstack 1.0.1