'nova rescue' fails if an instance image does not have a kernel_id

Bug #1028509 reported by Andrew Glen-Young
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Unassigned
nova (Ubuntu)
Medium
Unassigned

Bug Description

A client of mine attempted to rescue an instance by using the `nova rescue` command which failed putting the instance in the `ERROR` state.

I am running Openstack Essex on Ubuntu 12.04 with KVM as my hypervisor and I do not have `rescue_image_id`, `rescue_kernel_id` nor `rescue_ramdisk_id` defined in my nova.conf file.

The log (included below) indicates that the rescue failed to complete as the rescue kernel does not exist. This is expected as the config does not include the `rescue_*_id` variables and the instance was started with an image which does not include a ramdisk or kernel image.

What happens:
-------------

The user is informed that an error occurred and the instance is set to the `error` state.

What I expect:
--------------

If `rescue_kernel_id` is not defined in nova.conf and the instance does not have a valid kernel_id, then the user should be informed that the instance cannot be rescued rather than the instance going into the error state and becoming inaccessible.

Log file:
---------

2012-07-24 14:20:08 ERROR nova.compute.manager [req-2bdf6c17-1733-475f-b036-0e66a0bff266 7739264c2246454f9bdbc8a24ad30a63 cf5a8bcc6652400593b472ba82c2c2b5] unable to set user and group to '107:116' on '/srv/nova/instances/instance-0000367b/kernel.rescue': No such file or directory. Setting instance vm_state to ERROR
2012-07-24 14:20:08 ERROR nova.rpc.amqp [req-2bdf6c17-1733-475f-b036-0e66a0bff266 7739264c2246454f9bdbc8a24ad30a63 cf5a8bcc6652400593b472ba82c2c2b5] Exception during message handling
2012-07-24 14:20:08 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-07-24 14:20:08 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-07-24 14:20:08 TRACE nova.rpc.amqp return f(*args, **kw)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2012-07-24 14:20:08 TRACE nova.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-07-24 14:20:08 TRACE nova.rpc.amqp sys.exc_info())
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-07-24 14:20:08 TRACE nova.rpc.amqp self.gen.next()
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-07-24 14:20:08 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1133, in rescue_instance
2012-07-24 14:20:08 TRACE nova.rpc.amqp self._legacy_nw_info(network_info), image_meta)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-07-24 14:20:08 TRACE nova.rpc.amqp return f(*args, **kw)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 853, in rescue
2012-07-24 14:20:08 TRACE nova.rpc.amqp self._hard_reboot(instance, network_info, xml=xml)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 783, in _hard_reboot
2012-07-24 14:20:08 TRACE nova.rpc.amqp self._create_new_domain(xml)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1589, in _create_new_domain
2012-07-24 14:20:08 TRACE nova.rpc.amqp domain.createWithFlags(launch_flags)
2012-07-24 14:20:08 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
2012-07-24 14:20:08 TRACE nova.rpc.amqp if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-07-24 14:20:08 TRACE nova.rpc.amqp libvirtError: unable to set user and group to '107:116' on '/srv/nova/instances/instance-0000367b/kernel.rescue': No such file or directory

SQL showing the relevant image fields for the instance:
-------------------------------------------------------

mysql> select image_ref, kernel_id, ramdisk_id from instances where id = '13947';
+--------------------------------------+-----------+------------+
| image_ref | kernel_id | ramdisk_id |
+--------------------------------------+-----------+------------+
| 3400daaa-fbea-407b-b92c-5b66c6f168cf | | |
+--------------------------------------+-----------+------------+

System information:
-------------------

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"

$ dpkg-query --show 'nova*'
nova-api 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-cert 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-common 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-doc 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-objectstore 2012.1+stable~20120612-3ee026e-0ubuntu1.1
nova-scheduler 2012.1+stable~20120612-3ee026e-0ubuntu1.1

description: updated
description: updated
Dave Walker (davewalker)
Changed in nova (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers