OpenStack Compute (nova)

  1. Bug #1584630
  2. Activity log

Activity log for bug #1584630

Date Who What changed Old value New value Message
2016-05-23 07:51:22 Jiajun Liu bug added bug
2016-05-23 07:52:22 Jiajun Liu description Steps to reproduce =================== 1, launch a boot from volume instance named bfv1 2, attach a volume (blank-volume1) to instance bfv1 3, take a snapshot on bfv1 and a new image bfv1-snap1 created 4, launch another boot from image instance named bfi1 4, rebuild bfi1 from bfv1-snap1 failed 5, error log is as follow 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher payload) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._rebuild_default_impl(**kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher block_device_info=new_block_device_info) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher admin_pass=admin_password) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher instance, size, fallback_from_host) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher size=size) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self.driver.import_image(base, self.rbd_name) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher utils.execute('rbd', 'import', *args) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command. 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u'' 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n" Environment ============ * os: centos7 * openstack: stable/liberty I looked at the master code and it have the same bug. Steps to reproduce =================== 1, launch a boot from volume instance named bfv1 2, attach a volume (blank-volume1) to instance bfv1 3, take a snapshot on bfv1 and a new image bfv1-snap1 created 4, launch another boot from image instance named bfi1 4, rebuild bfi1 from bfv1-snap1 failed 5, error log is as follow 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher payload) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__ 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher bdms, recreate, on_shared_storage, preserve_ephemeral) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._do_rebuild_instance(*args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self._rebuild_default_impl(**kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher block_device_info=new_block_device_info) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher admin_pass=admin_password) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher instance, size, fallback_from_host) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher size=size) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher *args, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher self.driver.import_image(base, self.rbd_name) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher utils.execute('rbd', 'import', *args) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd) 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command. 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u'' 2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n" Environment ============  * os: centos7  * openstack: stable/liberty * storage: ceph rbd I looked at the master code and it have the same bug.
2016-05-24 13:14:13 Markus Zoeller (markus_z) tags ceph
2016-06-04 01:38:57 Matt Riedemann tags ceph ceph rebuild volumes
2016-06-08 03:07:11 zehua nova: assignee zehua (zehua)
2016-06-08 08:02:14 Takashi Natsume nova: status New In Progress
2016-10-13 13:53:02 Dr. Jens Harbott bug added subscriber Dr. Jens Rosenboom
2017-02-15 23:14:46 Matt Riedemann nova: assignee zehua (zehua)
2017-02-15 23:14:50 Matt Riedemann nova: status In Progress Confirmed
2017-06-27 15:55:37 Sean Dague tags ceph rebuild volumes ceph openstack-version.liberty rebuild volumes