OpenStack Compute (nova)

  1. Bug #1356534
  2. Activity log

Activity log for bug #1356534

Date Who What changed Old value New value Message
2014-08-13 19:11:46 Mike Dorman bug added bug
2014-08-13 19:16:12 Mike Dorman description When force_config_drive=Always is set, VMs that did not previously have a config drive created for them will fail to start. In our particular use case, we had NOT been using config drive for a while, and then enabled it with force_config_drive=Always. Any VMs created before that time did not have a config drive created, and are now failing to start because nova-compute expects all VMs to have one. 2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp **args) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp payload) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp pass 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._power_on(context, instance) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._hard_reboot(context, instance, network_info, block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp dk_size = int(os.path.getsize(path)) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return os.stat(filename).st_size 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config' 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp The work around for now is to just create a 0-byte file (or empty iso image) for disk.config on the VMs that didn't previously have one. But, in our opinion, nova should not fail to boot the VM in this situation. I see there as two valid behaviors: 1. Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM 2. Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one. When force_config_drive=Always is set, VMs that did not previously have a config drive created for them will fail to start. In our particular use case, we had NOT been using config drive for a while, and then enabled it with force_config_drive=Always. Any VMs created before that time did not have a config drive created, and are now failing to start because nova-compute expects all VMs to have one. 2014-08-13 11:32:22.459 4711 ERROR nova.openstack.common.rpc.amqp [req-3d24e130-a682-415f-a6be-c3e9f3e97e39 02d7755lyxlnA 1be95d2dfcae4ab281004e22553c0d92] Exception during message handling 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp **args) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 353, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp payload) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp pass 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 294, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1853, in start_instance 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._power_on(context, instance) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1840, in _power_on 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1969, in power_on 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp self._hard_reboot(context, instance, network_info, block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1924, in _hard_reboot 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp block_device_info) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4380, in get_instance_disk_info 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp dk_size = int(os.path.getsize(path)) 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp return os.stat(filename).st_size 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/8625b6cf-2ab2-4a05-8e15-ae834f250393/disk.config' 2014-08-13 11:32:22.459 4711 TRACE nova.openstack.common.rpc.amqp The work around for now is to just create a 0-byte file (or empty iso image) for disk.config on the VMs that didn't previously have one. But, in our opinion, nova should not fail to boot the VM in this situation. I see there as three valid behaviors: 1. Recognize that the config drive is missing, but ignore it and do not attach a config drive device to the VM 2. Retroactively create the config drive disk for the VM if it is attempting to start and does not already have one. 3. Recreate the config drive disk for VMs on every boot. This has the added advantage of making the config drive dynamic.
2014-08-26 17:19:18 Mike Dorman attachment added nova-compute.log https://bugs.launchpad.net/nova/+bug/1356534/+attachment/4187526/+files/nova-compute.log
2014-08-27 16:43:58 Tracy Jones tags compute
2014-09-10 13:32:28 Sean Dague nova: status New Confirmed
2014-09-10 13:32:33 Sean Dague nova: importance Undecided Low
2014-09-10 13:33:11 Sean Dague nova: status Confirmed Triaged
2014-09-10 13:33:26 Sean Dague tags compute compute low-hanging-fruit
2014-10-11 01:49:58 Jean-Daniel Bussy nova: assignee BUSSY Jean-Daniel (silversurfer972)
2014-11-02 13:05:04 Marco Pavone bug added subscriber Marco Pavone
2015-03-21 07:41:20 wuhao nova: assignee Jean-Daniel Bussy (silversurfer972) wuhao (wuhao)
2015-03-21 07:47:34 OpenStack Infra nova: status Triaged In Progress
2015-09-04 16:30:59 John Garbutt nova: importance Low Medium
2015-09-04 16:31:02 John Garbutt nova: status In Progress Triaged
2015-09-04 16:31:05 John Garbutt nova: assignee wuhao (wuhao)
2015-09-04 16:31:11 John Garbutt tags compute low-hanging-fruit compute libvirt low-hanging-fruit
2015-09-10 07:57:16 Zhenzan Zhou nova: assignee Zhenzan Zhou (zhenzan-zhou)
2015-09-11 05:25:10 OpenStack Infra nova: status Triaged In Progress
2015-09-14 09:38:38 Abhijeet Malawade bug added subscriber Abhijeet Malawade
2015-09-28 06:55:54 Zhenzan Zhou marked as duplicate 1241806