cells: test_create_ebs_image_and_check_boot fails intermittently

[Andrew Laski]

2015-08-28 18:41:12.128 | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_create_ebs_image_and_check_boot[compute,id-36c34c67-7b54-4b59-b188-02a2f458a63b,image,volume]
2015-08-28 18:41:12.128 | --------------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-08-28 18:41:12.128 |
2015-08-28 18:41:12.128 | Captured traceback:
2015-08-28 18:41:12.128 | ~~~~~~~~~~~~~~~~~~~
2015-08-28 18:41:12.128 | Traceback (most recent call last):
2015-08-28 18:41:12.128 | File "tempest/test.py", line 126, in wrapper
2015-08-28 18:41:12.128 | return f(self, *func_args, **func_kwargs)
2015-08-28 18:41:12.128 | File "tempest/scenario/test_volume_boot_pattern.py", line 194, in test_create_ebs_image_and_check_boot
2015-08-28 18:41:12.128 | instance = self.create_server(image=image['id'])
2015-08-28 18:41:12.129 | File "tempest/scenario/manager.py", line 177, in create_server
2015-08-28 18:41:12.129 | **create_kwargs)
2015-08-28 18:41:12.129 | File "tempest/services/compute/json/servers_client.py", line 86, in create_server
2015-08-28 18:41:12.129 | resp, body = self.post('servers', post_body)
2015-08-28 18:41:12.129 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 256, in post
2015-08-28 18:41:12.129 | return self.request('POST', url, extra_headers, headers, body)
2015-08-28 18:41:12.129 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 643, in request
2015-08-28 18:41:12.129 | resp, resp_body)
2015-08-28 18:41:12.129 | File "/opt/stack/new/tempest/.tox/all/local/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 700, in _error_checker
2015-08-28 18:41:12.129 | raise exceptions.BadRequest(resp_body)
2015-08-28 18:41:12.129 | tempest_lib.exceptions.BadRequest: Bad request
2015-08-28 18:41:12.130 | Details: {u'code': 400, u'message': u'Block Device Mapping is Invalid: Boot sequence for the instance and image/block device mapping combination is not valid.'}

as seen at http://logs.openstack.org/96/216696/6/check/gate-tempest-dsvm-cells/5304a42/console.html

The cause appears to be a race with processing messages to create_or_update a bdm in the parent cell. Rather than creating then updating the two messages both create due to the quickness with which they are sent and processed. One solution is to send a create message and then an update. A followup to enhance that would include adding a unique constraint so that two identical bdms can't be created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/218450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218450
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5e7dc40041d2870560b96b48f90a75f26642a234
Submitter: Jenkins
Branch: master

commit 5e7dc40041d2870560b96b48f90a75f26642a234
Author: Andrew Laski <email address hidden>
Date: Fri Aug 28 15:41:05 2015 -0400

    Cells: Improve block device mapping update/create calls

    There is a single RPC method used to both create and update block device
    mappings in a parent cell. Based on arguments passed to it it will
    either create/update/update_or_create the block device mapping. This
    could cause duplicate bdms to be created if multiple update_or_create
    calls come in quickly. Instead the update_or_create argument should
    only be used when sending the first RPC message to the parent cell.

    There's additional complexity here in that a bdm object create() call
    should create the bdm and all save() calls should update. However
    create() doesn't send an update message if device_name isn't set due to
    uniqueness being determined by a combination of instance uuid and device
    name. So save() needs to use update_or_create if device name is being
    changed, and update all other times.

    Change-Id: Ic093f003abfbfe00500c318f7ee0abe884394342
    Closes-bug: 1490011