Hi there,
I'm using a PackStack with Newton.
What I do:
1) Create a Heat stack with this template:
heat_template_version: 2015-04-30
description: Simple template to deploy a single compute instance
resources:
server:
type: OS::Nova::Server
properties:
image: cirros
flavor: m1.tiny
networks:
- network: 770ec23b-6198-47aa-8aa2-310de3fe754a
2) I snapshot the stack via 'openstack stack snapshot create stack1'
3) I update the stack with this template:
heat_template_version: 2015-04-30
description: Simple template to deploy a single compute instance
resources:
server:
type: OS::Nova::Server
properties:
image: cirros
flavor: m1.tiny
networks:
- network: 770ec23b-6198-47aa-8aa2-310de3fe754a
cinder_volume:
type: OS::Cinder::Volume
properties:
size: 1
volume_attachment:
type: OS::Cinder::VolumeAttachment
properties:
volume_id: { get_resource: cinder_volume }
instance_uuid: { get_resource: server }
mountpoint: /dev/sdb
4) I wait until it is finished successfully, then restore the snapshot via 'openstack stack snapshot restore...'
At this point, it fails. 'openstack stack show' tells me:
Error: resources.server: Rebuilding server failed, status 'ERROR'
Looking into 'openstack server show':
{u'message': u'Block Device Mapping is Invalid.', u'code': 400, u'created': u'2017-10-30T09:56:29Z'}
Heat logs do not contain much information.
nova-compute.log:
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1586, in _prep_block_device
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] wait_func=self._await_block_device_map_created)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 514, in attach_block_devices
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] map(_log_and_attach, block_device_mapping)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 512, in _log_and_attach
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] bdm.attach(*attach_args, **attach_kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 51, in wrapped
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] obj.save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 349, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] super(DriverVolumeBlockDevice, self).save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 162, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._bdm_obj.save()
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] ctxt, self, fn.__name__, args, kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_action
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] objmethod=objmethod, args=args, kwargs=kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] retry=self.retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] timeout=timeout, retry=retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] retry=retry)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise result
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] BDMNotFound_Remote: No Block Device Mapping with id 30.
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 87, in _object_dispatch
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] return getattr(target, method)(*args, **kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] return fn(self, *args, **kwargs)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 193, in save
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise exception.BDMNotFound(id=self.id)
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390]
2017-10-30 10:56:29.101 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] BDMNotFound: No Block Device Mapping with id 30.
...
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [req-1e35c505-0d66-4c1c-a23c-76301a80cf26 6d3a7dbdccb9489dba29722e1bc85770 af665cfb27724f3a8405880f142157ab - - -] [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Setting instance vm_state to ERROR
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] Traceback (most recent call last):
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6612, in _error_out_instance_on_exception
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] yield
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2705, in rebuild_instance
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2749, in _do_rebuild_instance_with_claim
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._do_rebuild_instance(*args, **kwargs)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2864, in _do_rebuild_instance
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] self._rebuild_default_impl(**kwargs)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2608, in _rebuild_default_impl
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] new_block_device_info = attach_block_devices(context, instance, bdms)
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1600, in _prep_block_device
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] raise exception.InvalidBDM()
2017-10-30 10:56:29.109 2333 ERROR nova.compute.manager [instance: 1979cfaf-1fdf-4156-aeed-eef04f081390] InvalidBDM: Block Device Mapping is Invalid.
...
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server [req-1e35c505-0d66-4c1c-a23c-76301a80cf26 6d3a7dbdccb9489dba29722e1bc85770 af665cfb27724f3a8405880f142157ab - - -] Exception during message handling
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2705, in rebuild_instance
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server bdms, recreate, on_shared_storage, preserve_ephemeral)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2749, in _do_rebuild_instance_with_claim
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self._do_rebuild_instance(*args, **kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2864, in _do_rebuild_instance
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server self._rebuild_default_impl(**kwargs)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2608, in _rebuild_default_impl
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server new_block_device_info = attach_block_devices(context, instance, bdms)
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1600, in _prep_block_device
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server raise exception.InvalidBDM()
2017-10-30 10:56:29.521 2333 ERROR oslo_messaging.rpc.server InvalidBDM: Block Device Mapping is Invalid.
It can't be reproduced on my computer. allocate_ retries, block_device_ allocate_ retries_ interval in nova.conf. And then check the volume list whether there is a lastest created volume.
Please check the parameters such as block_device_