OpenStack Compute (nova)

  1. Bug #1522667
  2. Comment #1

Comment 1 for bug 1522667

Revision history for this message
Michael Still (mikal) wrote : Re: when nova-compute.service start .If the instance's disk is not avilable.It will cause the service failed.That is not reasonable

Despite this mangled traceback, I was able to re-create this with devstack on a public cloud instance. I followed these steps:

- boot an instance
- stop the instance (nova stop)
- remove the instance's root disk from /opt/stack/data/nova/instances/...
- start the instance (nova start)

This is what I get:

2015-12-07 21:59:20.622 ERROR oslo_messaging.rpc.dispatcher [req-cadd058d-3ac7-424e-9e97-8386c8afc079 admin demo] Exception during message handling: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/f9a712f1-bd23-426c-846d-3d4cda57e342/disk'
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 105, in wrapped
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher payload)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 349, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 322, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377,
 in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/e
xcutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365,
 in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2534, in start_instance
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2507, in _power_on
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2360, in power_on
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher self._hard_reboot(context, instance, network_info, block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2224, in _hard_reboot
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6556, in _get_instance_disk_info
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher dk_size = int(os.path.getsize(path))
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return os.stat(filename).st_size
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher OSError: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/f9a712f1-bd23-426c-846d-3d4cda57e342/disk'
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher

I also don't love that we don't put the instance into an error state when this happens:

$ nova list
+--------------------------------------+------+---------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+------+---------+------------+-------------+------------------+
| f9a712f1-bd23-426c-846d-3d4cda57e342 | foo | SHUTOFF | - | Shutdown | private=10.0.0.2 |
 +--------------------------------------+------+---------+------------+-------------+------------------+