Confirmed that the issue is when nova is asked to create a snapshot of the instance with the volume attached, it's actually creating 2 volume snapshots, and those are the bdms that go into the image snapshot metadata. Then later when we boot the instance from that image, it has 2 bdms in the image metadata both with boot_index=0 and that fails.
So we're probably hitting cells.messaging.bdm_update_or_create_at_top twice and creating a new bdm each time, even though we should be updating the second time, not doing a create.
Notes from logs:
first server id: 2c9cecc1-c3db-4057-81bd-98e488c45ac2
2015-11-03 21:45:08.411 INFO cinder.volume.api [req-db073638-35db-46c5-a358-d832113fc3ef None] Snapshot(cgsnapshot=<?>,cgsnapshot_id=None,created_at=2015-11-03T21:45:08Z,deleted=False,deleted_at=None,display_description=None,display_name='snapshot for tempest-image-1751811273',encryption_key_id=None,id=9b4fe063-6427-4000-a286-7369a7677a17,metadata={},progress='0%',project_id=a86353192f7b42539f247f284249e0bf,provider_auth=None,provider_id=None,provider_location=None,status='creating',updated_at=None,user_id=2e821b0f84a9402d91463d8a7a2d4de7,volume=<?>,volume_id=35909d21-81b8-4fda-82b6-d3d75be61238,volume_size=1,volume_type_id=09c3a4dd-e7b1-4cd1-a03d-809216ba9b44)Snapshot force create request issued successfully.
2015-11-03 21:45:08.925 INFO cinder.volume.api [req-eebfe355-197d-4e01-a946-41f05f733884 None] Snapshot(cgsnapshot=<?>,cgsnapshot_id=None,created_at=2015-11-03T21:45:08Z,deleted=False,deleted_at=None,display_description=None,display_name='snapshot for tempest-image-1751811273',encryption_key_id=None,id=f01c94bc-ddac-4687-8620-41eeacf57b0f,metadata={},progress='0%',project_id=a86353192f7b42539f247f284249e0bf,provider_auth=None,provider_id=None,provider_location=None,status='creating',updated_at=None,user_id=2e821b0f84a9402d91463d8a7a2d4de7,volume=<?>,volume_id=35909d21-81b8-4fda-82b6-d3d75be61238,volume_size=1,volume_type_id=09c3a4dd-e7b1-4cd1-a03d-809216ba9b44)Snapshot force create request issued successfully.
Confirmed that the issue is when nova is asked to create a snapshot of the instance with the volume attached, it's actually creating 2 volume snapshots, and those are the bdms that go into the image snapshot metadata. Then later when we boot the instance from that image, it has 2 bdms in the image metadata both with boot_index=0 and that fails.
So we're probably hitting cells.messaging .bdm_update_ or_create_ at_top twice and creating a new bdm each time, even though we should be updating the second time, not doing a create.
Notes from logs:
first server id: 2c9cecc1- c3db-4057- 81bd-98e488c45a c2
snapshot 1 request to cinder:
http:// logs.openstack. org/66/ 241366/ 1/check/ gate-tempest- dsvm-cells/ 1d7551e/ logs/screen- c-api.txt. gz#_2015- 11-03_21_ 45_08_411
2015-11-03 21:45:08.411 INFO cinder.volume.api [req-db073638- 35db-46c5- a358-d832113fc3 ef None] Snapshot( cgsnapshot= <?>,cgsnapshot_ id=None, created_ at=2015- 11-03T21: 45:08Z, deleted= False,deleted_ at=None, display_ description= None,display_ name='snapshot for tempest- image-175181127 3',encryption_ key_id= None,id= 9b4fe063- 6427-4000- a286-7369a7677a 17,metadata= {},progress= '0%',project_ id=a86353192f7b 42539f247f28424 9e0bf,provider_ auth=None, provider_ id=None, provider_ location= None,status= 'creating' ,updated_ at=None, user_id= 2e821b0f84a9402 d91463d8a7a2d4d e7,volume= <?>,volume_ id=35909d21- 81b8-4fda- 82b6-d3d75be612 38,volume_ size=1, volume_ type_id= 09c3a4dd- e7b1-4cd1- a03d-809216ba9b 44)Snapshot force create request issued successfully.
snapshot 2 request to cinder (same volume_id):
http:// logs.openstack. org/66/ 241366/ 1/check/ gate-tempest- dsvm-cells/ 1d7551e/ logs/screen- c-api.txt. gz#_2015- 11-03_21_ 45_08_925
2015-11-03 21:45:08.925 INFO cinder.volume.api [req-eebfe355- 197d-4e01- a946-41f05f7338 84 None] Snapshot( cgsnapshot= <?>,cgsnapshot_ id=None, created_ at=2015- 11-03T21: 45:08Z, deleted= False,deleted_ at=None, display_ description= None,display_ name='snapshot for tempest- image-175181127 3',encryption_ key_id= None,id= f01c94bc- ddac-4687- 8620-41eeacf57b 0f,metadata= {},progress= '0%',project_ id=a86353192f7b 42539f247f28424 9e0bf,provider_ auth=None, provider_ id=None, provider_ location= None,status= 'creating' ,updated_ at=None, user_id= 2e821b0f84a9402 d91463d8a7a2d4d e7,volume= <?>,volume_ id=35909d21- 81b8-4fda- 82b6-d3d75be612 38,volume_ size=1, volume_ type_id= 09c3a4dd- e7b1-4cd1- a03d-809216ba9b 44)Snapshot force create request issued successfully.
--
image snapshot get response:
http:// logs.openstack. org/66/ 241366/ 1/check/ gate-tempest- dsvm-cells/ 1d7551e/ console. html#_2015- 11-03_21_ 50_38_475
'[{"guest_format": null, "boot_index": 0, "delete_ on_termination" : true, "no_device": null, "snapshot_id": "9b4fe063- 6427-4000- a286-7369a7677a 17", "device_name": "/dev/vda", "disk_bus": "virtio", "image_id": null, "source_type": "snapshot", "device_type": "disk", "volume_id": null, "destination_type": "volume", "volume_size": 1}, {"guest_format": null, "boot_index": 0, "delete_ on_termination" : true, "no_device": null, "snapshot_id": "f01c94bc- ddac-4687- 8620-41eeacf57b 0f", "device_name": "/dev/vda", "disk_bus": "virtio", "image_id": null, "source_type": "snapshot", "device_type": "disk", "volume_id": null, "destination_type": "volume", "volume_size": 1}]'
400 on invalid bdm sequence:
2015-11-03 21:45:18.449 DEBUG nova.compute.api [req-f4c02d3a- b257-4e97- 992d-54f5f56a41 4f tempest- TestVolumeBootP atternV2- 2140390147 tempest- TestVolumeBootP atternV2- 999967296] [instance: b4d6d91b- a94c-45d9- 80dc-3e40d83afc ee] Invalid block device mapping boot sequence for instance: [BlockDeviceMap ping(boot_ index=0, connection_ info=None, created_ at=<?>, delete_ on_termination= True,deleted= <?>,deleted_ at=<?>, destination_ type='volume' ,device_ name='/ dev/vda' ,device_ type='disk' ,disk_bus= 'virtio' ,guest_ format= None,id= <?>,image_ id=None, instance= <?>,instance_ uuid=<? >,no_device= False,snapshot_ id='9b4fe063- 6427-4000- a286-7369a7677a 17',source_ type='snapshot' ,updated_ at=<?>, volume_ id=None, volume_ size=1) , BlockDeviceMapp ing(boot_ index=0, connection_ info=None, created_ at=<?>, delete_ on_termination= True,deleted= <?>,deleted_ at=<?>, destination_ type='volume' ,device_ name='/ dev/vda' ,device_ type='disk' ,disk_bus= 'virtio' ,guest_ format= None,id= <?>,image_ id=None, instance= <?>,instance_ uuid=<? >,no_device= False,snapshot_ id='f01c94bc- ddac-4687- 8620-41eeacf57b 0f',source_ type='snapshot' ,updated_ at=<?>, volume_ id=None, volume_ size=1) ] _validate_bdm /opt/stack/ new/nova/ nova/compute/ api.py: 1300