VM reboot fails with EC 42 (natty)

[Abel Lopez]

This is on natty with libvirt-bin 0.8.8-1ubuntu6

We observe random VM instances where a user initiated reboot causes the instance to fail to boot.
logs show:
2012-08-03 11:39:25,930 WARNING nova.virt.libvirt_conn [-] Error from libvirt during undefine of instance-00000191. Code=42 Error=Domain not found: no domain with matching uuid '2415b307-96d7-c779-f167-649f853f73a4'
2012-08-03 11:39:25,930 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 490, in reboot
(nova.exception): TRACE: self.destroy(instance, network_info, cleanup=False)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 299, in destroy
(nova.exception): TRACE: virt_dom.undefine()
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 745, in undefine
(nova.exception): TRACE: if ret == -1: raise libvirtError ('virDomainUndefine() failed', dom=self)
(nova.exception): TRACE: libvirtError: Domain not found: no domain with matching uuid '2415b307-96d7-c779-f167-649f853f73a4'
(nova.exception): TRACE:
2012-08-03 11:39:25,960 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 605, in reboot_instance
(nova.rpc): TRACE: self.driver.reboot(instance_ref, network_info)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: Domain not found: no domain with matching uuid '2415b307-96d7-c779-f167-649f853f73a4'
(nova.rpc): TRACE:

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

The next time this happens, could you confirm whether a domain identified by the specified uuid exists? This sounds more like a bug in nova, using the wrong uuid.

[Abel Lopez (al592b)]

Seeing this on another system: logs as follow:
 TRACE: libvirtError: Domain not found: no domain with matching name 'instance-000004d4'
TRACE: Error: Domain not found: no domain with matching name 'instance-000004d4'

File exists:
/var/lib/nova/instances/instance-000004d4# grep 4d4 libvirt.xml
    <name>instance-000004d4</name>
            <source file='/var/lib/nova/instances/instance-000004d4/disk'/>

Instance is seemingly undefined:

virsh list
 Id Name State
----------------------------------
  2 instance-000006e6 running
  3 instance-00000417 running
  4 instance-0000059b running

If I 'virsh define' using the file, I can resurrect the instance.

[Serge Hallyn (serge-hallyn)]

virsh list only shows running VMs. In such a case does 'virsh list --all' show the supposedly undefined domain?

I'm going to mark this as affecting nova as well. It could be a race in libvirt causing it to lose info, or it could be a mis-use of the libvirt api.

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]