Despite this mangled traceback, I was able to re-create this with devstack on a public cloud instance. I followed these steps:
- boot an instance
- stop the instance (nova stop)
- remove the instance's root disk from /opt/stack/data/nova/instances/...
- start the instance (nova start)
This is what I get:
2015-12-07 21:59:20.622 ERROR oslo_messaging.rpc.dispatcher [req-cadd058d-3ac7-424e-9e97-8386c8afc079 admin demo] Exception during message handling: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/f9a712f1-bd23-426c-846d-3d4cda57e342/disk'
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 105, in wrapped
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher payload)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 349, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 322, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377,
in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/e
xcutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365,
in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2534, in start_instance
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2507, in _power_on
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2360, in power_on
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher self._hard_reboot(context, instance, network_info, block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2224, in _hard_reboot
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6556, in _get_instance_disk_info
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher dk_size = int(os.path.getsize(path))
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher return os.stat(filename).st_size
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher OSError: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/f9a712f1-bd23-426c-846d-3d4cda57e342/disk'
2015-12-07 21:59:20.622 TRACE oslo_messaging.rpc.dispatcher
I also don't love that we don't put the instance into an error state when this happens:
$ nova list
+--------------------------------------+------+---------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+---------+------------+-------------+------------------+
| f9a712f1-bd23-426c-846d-3d4cda57e342 | foo | SHUTOFF | - | Shutdown | private=10.0.0.2 |
+--------------------------------------+------+---------+------------+-------------+------------------+
Despite this mangled traceback, I was able to re-create this with devstack on a public cloud instance. I followed these steps:
- boot an instance data/nova/ instances/ ...
- stop the instance (nova stop)
- remove the instance's root disk from /opt/stack/
- start the instance (nova start)
This is what I get:
2015-12-07 21:59:20.622 ERROR oslo_messaging. rpc.dispatcher [req-cadd058d- 3ac7-424e- 9e97-8386c8afc0 79 admin demo] Exception during message handling: [Errno 2] No such file or directory: '/opt/stack/ data/nova/ instances/ f9a712f1- bd23-426c- 846d-3d4cda57e3 42/disk' rpc.dispatcher Traceback (most recent call last): rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ rpc/dispatcher. py", line 142, in _dispatch_and_reply rpc.dispatcher executor_callback)) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ rpc/dispatcher. py", line 186, in _dispatch rpc.dispatcher executor_callback) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ rpc/dispatcher. py", line 129, in _do_dispatch rpc.dispatcher result = func(ctxt, **new_args) rpc.dispatcher File "/opt/stack/ nova/nova/ exception. py", line 105, in wrapped rpc.dispatcher payload) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 204, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ exception. py", line 88, in wrapped rpc.dispatcher return f(self, context, *args, **kw) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 349, in decorated_function rpc.dispatcher LOG.warning(msg, e, instance=instance) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 204, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 322, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 399, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 377, rpc.dispatcher kwargs['instance'], e, sys.exc_info()) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ e rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 365, rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 2534, in start_instance rpc.dispatcher self._power_ on(context, instance) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 2507, in _power_on rpc.dispatcher block_device_info) rpc.dispatcher File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 2360, in power_on rpc.dispatcher self._hard_ reboot( context, instance, network_info, block_device_info) rpc.dispatcher File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 2224, in _hard_reboot rpc.dispatcher block_device_info) rpc.dispatcher File "/opt/stack/ nova/nova/ virt/libvirt/ driver. py", line 6556, in _get_instance_ disk_info rpc.dispatcher dk_size = int(os. path.getsize( path)) rpc.dispatcher File "/usr/lib/ python2. 7/genericpath. py", line 49, in getsize rpc.dispatcher return os.stat( filename) .st_size rpc.dispatcher OSError: [Errno 2] No such file or directory: '/opt/stack/ data/nova/ instances/ f9a712f1- bd23-426c- 846d-3d4cda57e3 42/disk' rpc.dispatcher
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
xcutils.py", line 204, in __exit__
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
in decorated_function
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
2015-12-07 21:59:20.622 TRACE oslo_messaging.
I also don't love that we don't put the instance into an error state when this happens:
$ nova list ------- ------- ------- ------- ----+-- ----+-- ------- +------ ------+ ------- ------+ ------- ------- ----+ ------- ------- ------- ------- ----+-- ----+-- ------- +------ ------+ ------- ------+ ------- ------- ----+ bd23-426c- 846d-3d4cda57e3 42 | foo | SHUTOFF | - | Shutdown | private=10.0.0.2 | ------- ------- ------- ------- ----+-- ----+-- ------- +------ ------+ ------- ------+ ------- ------- ----+
+------
| ID | Name | Status | Task State | Power State | Networks |
+------
| f9a712f1-
+------