When force_config_drive=Always is set, VMs that did not previously have a config drive created for them will fail to start.
In our particular use case, we had NOT been using config drive for a while, and then enabled it with force_config_drive=Always. Any VMs created before that time did not have a config drive created, and are now failing to start because nova-compute expects all VMs to have one.
2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp **args)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp payload)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp pass
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._power_on(context, instance)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._hard_reboot(context, instance, network_info, block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info)
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp dk_size = int(os.path.getsize(path))
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return os.stat(filename).st_size
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config'
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp
The work around for now is to just create a 0-byte file (or empty iso image) for disk.config on the VMs that didn't previously have one.
But, in our opinion, nova should not fail to boot the VM in this situation. I see there as two valid behaviors:
1. Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM
2. Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one.
When force_config_ drive=Always is set, VMs that did not previously have a config drive created for them will fail to start.
In our particular use case, we had NOT been using config drive for a while, and then enabled it with force_config_ drive=Always. Any VMs created before that time did not have a config drive created, and are now failing to start because nova-compute expects all VMs to have one.
2014-08-13 11:32:22.459 4711 ERROR nova.openstack. common. rpc.amqp [req-3d24e130- a682-415f- a6be-c3e9f3e97e 39 02d7755lyxlnA 1be95d2dfcae4ab 281004e22553c0d 92] Exception during message handling common. rpc.amqp Traceback (most recent call last): common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/openstack/ common/ rpc/amqp. py", line 461, in _process_data common. rpc.amqp **args) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/openstack/ common/ rpc/dispatcher. py", line 172, in dispatch common. rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 353, in decorated_function common. rpc.amqp return function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/exception. py", line 90, in wrapped common. rpc.amqp payload) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/exception. py", line 73, in wrapped common. rpc.amqp return f(self, context, *args, **kw) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 243, in decorated_function common. rpc.amqp pass common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 229, in decorated_function common. rpc.amqp return function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 294, in decorated_function common. rpc.amqp function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 271, in decorated_function common. rpc.amqp e, sys.exc_info()) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 258, in decorated_function common. rpc.amqp return function(self, context, *args, **kwargs) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 1853, in start_instance common. rpc.amqp self._power_ on(context, instance) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 1840, in _power_on common. rpc.amqp block_device_info) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 1969, in power_on common. rpc.amqp self._hard_ reboot( context, instance, network_info, block_device_info) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 1924, in _hard_reboot common. rpc.amqp block_device_info) common. rpc.amqp File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 4380, in get_instance_ disk_info common. rpc.amqp dk_size = int(os. path.getsize( path)) common. rpc.amqp File "/usr/lib64/ python2. 6/genericpath. py", line 49, in getsize common. rpc.amqp return os.stat( filename) .st_size common. rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/ nova/instances/ 8625b6cf- 2ab2-4a05- 8e15-ae834f2503 93/disk. config' common. rpc.amqp
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
2014-08-13 11:32:22.459 4711 TRACE nova.openstack.
The work around for now is to just create a 0-byte file (or empty iso image) for disk.config on the VMs that didn't previously have one.
But, in our opinion, nova should not fail to boot the VM in this situation. I see there as two valid behaviors:
1. Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM
2. Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one.