OpenStack Compute (nova)

rebuild don't respect image's block device mapping propertries

Bug #1584630 reported by Jiajun Liu on 2016-05-23

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	Undecided	Unassigned

Bug Description

Steps to reproduce
===================

1, launch a boot from volume instance named bfv1
2, attach a volume (blank-volume1) to instance bfv1
3, take a snapshot on bfv1 and a new image bfv1-snap1 created
4, launch another boot from image instance named bfi1
4, rebuild bfi1 from bfv1-snap1 failed
5, error log is as follow

2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher payload)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._rebuild_default_impl(**kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher block_device_info=new_block_device_info)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher admin_pass=admin_password)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher instance, size, fallback_from_host)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher size=size)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self.driver.import_image(base, self.rbd_name)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher utils.execute('rbd', 'import', *args)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n"

Environment
============
* os: centos7
* openstack: stable/liberty
* storage: ceph rbd

I looked at the master code and it have the same bug.

See original description

Tags:

Jiajun Liu (ljjjustin) on 2016-05-23

description:

updated

Markus Zoeller (markus_z) (mzoeller) on 2016-05-24

tags:

added: ceph

Matt Riedemann (mriedem) on 2016-06-04

tags:

added: rebuild volumes

zehua (zehua) on 2016-06-08

Changed in nova:
assignee:	nobody → zehua (zehua)

Takashi Natsume (natsume-takashi) on 2016-06-08

Changed in nova:
status:	New → In Progress

Revision history for this message

zehua (zehua) wrote on 2016-07-12:

Should we replace all origin volume by new volume provided from image block_device_mapping attribute?

Matt Riedemann (mriedem) on 2017-02-15

Changed in nova:
assignee:	zehua (zehua) → nobody
status:	In Progress → Confirmed

Revision history for this message

Sean Dague (sdague) wrote on 2017-06-27:

Automatically discovered version liberty in description. If this is incorrect, please update the description to include 'nova version: ...'

tags:

added: openstack-version.liberty

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.