Confirmed that the issue is when nova is asked to create a snapshot of the instance with the volume attached, it's actually creating 2 volume snapshots, and those are the bdms that go into the image snapshot metadata. Then later when we boot the instance from that image, it has 2 bdms in the image metadata both with boot_index=0 and that fails.

So we're probably hitting cells.messaging.bdm_update_or_create_at_top twice and creating a new bdm each time, even though we should be updating the second time, not doing a create.

Notes from logs:

first server id: 2c9cecc1-c3db-4057-81bd-98e488c45ac2

snapshot 1 request to cinder:

http://logs.openstack.org/66/241366/1/check/gate-tempest-dsvm-cells/1d7551e/logs/screen-c-api.txt.gz#_2015-11-03_21_45_08_411

2015-11-03 21:45:08.411 INFO cinder.volume.api [req-db073638-35db-46c5-a358-d832113fc3ef None] Snapshot(cgsnapshot=<?>,cgsnapshot_id=None,created_at=2015-11-03T21:45:08Z,deleted=False,deleted_at=None,display_description=None,display_name='snapshot for tempest-image-1751811273',encryption_key_id=None,id=9b4fe063-6427-4000-a286-7369a7677a17,metadata={},progress='0%',project_id=a86353192f7b42539f247f284249e0bf,provider_auth=None,provider_id=None,provider_location=None,status='creating',updated_at=None,user_id=2e821b0f84a9402d91463d8a7a2d4de7,volume=<?>,volume_id=35909d21-81b8-4fda-82b6-d3d75be61238,volume_size=1,volume_type_id=09c3a4dd-e7b1-4cd1-a03d-809216ba9b44)Snapshot force create request issued successfully.

snapshot 2 request to cinder (same volume_id):

http://logs.openstack.org/66/241366/1/check/gate-tempest-dsvm-cells/1d7551e/logs/screen-c-api.txt.gz#_2015-11-03_21_45_08_925

2015-11-03 21:45:08.925 INFO cinder.volume.api [req-eebfe355-197d-4e01-a946-41f05f733884 None] Snapshot(cgsnapshot=<?>,cgsnapshot_id=None,created_at=2015-11-03T21:45:08Z,deleted=False,deleted_at=None,display_description=None,display_name='snapshot for tempest-image-1751811273',encryption_key_id=None,id=f01c94bc-ddac-4687-8620-41eeacf57b0f,metadata={},progress='0%',project_id=a86353192f7b42539f247f284249e0bf,provider_auth=None,provider_id=None,provider_location=None,status='creating',updated_at=None,user_id=2e821b0f84a9402d91463d8a7a2d4de7,volume=<?>,volume_id=35909d21-81b8-4fda-82b6-d3d75be61238,volume_size=1,volume_type_id=09c3a4dd-e7b1-4cd1-a03d-809216ba9b44)Snapshot force create request issued successfully.

--

image snapshot get response:

http://logs.openstack.org/66/241366/1/check/gate-tempest-dsvm-cells/1d7551e/console.html#_2015-11-03_21_50_38_475

'[{"guest_format": null, "boot_index": 0, "delete_on_termination": true, "no_device": null, "snapshot_id": "9b4fe063-6427-4000-a286-7369a7677a17", "device_name": "/dev/vda", "disk_bus": "virtio", "image_id": null, "source_type": "snapshot", "device_type": "disk", "volume_id": null, "destination_type": "volume", "volume_size": 1}, {"guest_format": null, "boot_index": 0, "delete_on_termination": true, "no_device": null, "snapshot_id": "f01c94bc-ddac-4687-8620-41eeacf57b0f", "device_name": "/dev/vda", "disk_bus": "virtio", "image_id": null, "source_type": "snapshot", "device_type": "disk", "volume_id": null, "destination_type": "volume", "volume_size": 1}]'

400 on invalid bdm sequence:

2015-11-03 21:45:18.449 DEBUG nova.compute.api [req-f4c02d3a-b257-4e97-992d-54f5f56a414f tempest-TestVolumeBootPatternV2-2140390147 tempest-TestVolumeBootPatternV2-999967296] [instance: b4d6d91b-a94c-45d9-80dc-3e40d83afcee] Invalid block device mapping boot sequence for instance: [BlockDeviceMapping(boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='volume',device_name='/dev/vda',device_type='disk',disk_bus='virtio',guest_format=None,id=<?>,image_id=None,instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id='9b4fe063-6427-4000-a286-7369a7677a17',source_type='snapshot',updated_at=<?>,volume_id=None,volume_size=1), BlockDeviceMapping(boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='volume',device_name='/dev/vda',device_type='disk',disk_bus='virtio',guest_format=None,id=<?>,image_id=None,instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id='f01c94bc-ddac-4687-8620-41eeacf57b0f',source_type='snapshot',updated_at=<?>,volume_id=None,volume_size=1)] _validate_bdm /opt/stack/new/nova/nova/compute/api.py:1300