OpenStack Heat

Invalid block device mapping after restoring stack snapshot

Bug #1728558 reported by David Rabel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Undecided
Unassigned
OpenStack Heat no-milestone-taged-bugs

Bug Description

Hi there,

I'm using a PackStack with Newton.

What I do:

1) Create a Heat stack with this template:
heat_template_version: 2015-04-30

description: Simple template to deploy a single compute instance

resources:
  server:
    type: OS::Nova::Server
    properties:
      image: cirros
      flavor: m1.tiny
      networks:
       - network: 770ec23b-6198-47aa-8aa2-310de3fe754a

2) I snapshot the stack via 'openstack stack snapshot create stack1'

3) I update the stack with this template:
heat_template_version: 2015-04-30

description: Simple template to deploy a single compute instance

resources:
  server:
    type: OS::Nova::Server
    properties:
      image: cirros
      flavor: m1.tiny
      networks:
       - network: 770ec23b-6198-47aa-8aa2-310de3fe754a
  cinder_volume:
    type: OS::Cinder::Volume
    properties:
      size: 1
  volume_attachment:
    type: OS::Cinder::VolumeAttachment
    properties:
      volume_id: { get_resource: cinder_volume }
      instance_uuid: { get_resource: server }
      mountpoint: /dev/sdb

4) I wait until it is finished successfully, then restore the snapshot via 'openstack stack snapshot restore...'

At this point, it fails. 'openstack stack show' tells me:
Error: resources.server: Rebuilding server failed, status 'ERROR'

Looking into 'openstack server show':
{u'message': u'Block Device Mapping is Invalid.', u'code': 400, u'created': u'2017-10-30T09:56:29Z'}

Heat logs do not contain much information.

nova-compute.log:

2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1586, in _prep_block_device
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] wait_func=self._await_block_device_map_created)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 514, in attach_block_devices
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] map(_log_and_attach, block_device_mapping)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 512, in _log_and_attach
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] bdm.attach(*attach_args, **attach_kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 51, in wrapped
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] obj.save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 349, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] super(DriverVolumeBlockDevice, self).save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 162, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._bdm_obj.save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] ctxt, self, fn.__name__, args, kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_action
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] objmethod=objmethod, args=args, kwargs=kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] retry=self.retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] timeout=timeout, retry=retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] retry=retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise result
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] BDMNotFound_Remote: No Block Device Mapping with id 30.
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 87, in _object_dispatch
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] return getattr(target, method)(*args, **kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] return fn(self, *args, **kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 193, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise exception.BDMNotFound(id=self.id)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] BDMNotFound: No Block Device Mapping with id 30.
...
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [req-1e35c505-0d66-4c1c-a23c-76301a80cf26 6d3a7dbdccb9489dba29722e1bc85770 af665cfb27724f3a8405880f142157ab - - -] [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Setting instance vm_state to ERROR
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6612, in _error_out_instance_on_exception
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] yield
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2705, in rebuild_instance
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2749, in _do_rebuild_instance_with_claim
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._do_rebuild_instance(*args, **kwargs)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2864, in _do_rebuild_instance
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._rebuild_default_impl(**kwargs)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2608, in _rebuild_default_impl
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] new_block_device_info = attach_block_devices(context, instance, bdms)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1600, in _prep_block_device
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise exception.InvalidBDM()
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] InvalidBDM: Block Device Mapping is Invalid.
...
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server [req-1e35c505-0d66-4c1c-a23c-76301a80cf26 6d3a7dbdccb9489dba29722e1bc85770 af665cfb27724f3a8405880f142157ab - - -] Exception during message handling
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2705, in rebuild_instance
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2749, in _do_rebuild_instance_with_claim
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self._do_rebuild_instance(*args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2864, in _do_rebuild_instance
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self._rebuild_default_impl(**kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2608, in _rebuild_default_impl
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server new_block_device_info = attach_block_devices(context, instance, bdms)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1600, in _prep_block_device
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server raise exception.InvalidBDM()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server InvalidBDM: Block Device Mapping is Invalid.

Revision history for this message
huangsm (huangsm) wrote :

It can't be reproduced on my computer.
Please check the parameters such as block_device_allocate_retries, block_device_allocate_retries_interval in nova.conf. And then check the volume list whether there is a lastest created volume.

huangsm (huangsm)
Changed in heat:
assignee: nobody → huangsm (huangsm)
Revision history for this message
David Rabel (rabel-b1) wrote :

#block_device_allocate_retries=60
#block_device_allocate_retries_interval=3

Both commented, so both are set to default values.

Yes, there is a volume:

+--------------------------------------+-----------------------------------+-----------+------+------------------------------+
| ID | Display Name | Status | Size | Attached to |
 +--------------------------------------+-----------------------------------+-----------+------+------------------------------+
| 47942952-0b73-49df-a6e0-34616ad963a0 | stack1-cinder_volume-oufmb5hjio7c | available | 1 | |

Revision history for this message
huangsm (huangsm) wrote :

Why the volume's status is 'available'? It should be 'attached' before step 4.
At other steps, the volume is not exist.

Revision history for this message
David Rabel (rabel-b1) wrote :

The volume is "in-use" before step 4, of course.

After step 4 it is still there, but in status available.

huangsm (huangsm)
Changed in heat:
assignee: huangsm (huangsm) → nobody
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.