libvirtError: Domain not found: no domain with matching name 'i-00000077'

[Jaroslav Pulchart]

I build latest Nova Essex from https://github.com/openstack/nova/commits/stable/essex (future 2012.1.3), because I need to use backported fixes from Folsom:
https://github.com/openstack/nova/commit/09217abddc07bd4fbaca6c300075df2c68ffedf7
and
https://github.com/openstack/nova/commit/b375b4f1131d54315bb9952fcf2eff363b3b29b1

With theses fixes I'm now able to restore correctly instances after host node reboot/failover, great!!!

But I experienced one issue. When some instance on host node is in "stopped" state the nova-compute service cannot start. In log is this error:

2012-09-27 12:49:36 DEBUG nova.virt.libvirt.volume [req-0c260b96-3d17-4633-aef8-fab3f2ca545f None None] Attaching local device /dev/nova-volumes/vol-00000040 as vda from (pid=35405) connect_volume /usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py:57
2012-09-27 12:49:37 CRITICAL nova [-] Domain not found: no domain with matching name 'i-00000077'
2012-09-27 12:49:37 TRACE nova Traceback (most recent call last):
2012-09-27 12:49:37 TRACE nova File "/opt/openstack/bin/nova-compute", line 49, in <module>
2012-09-27 12:49:37 TRACE nova service.wait()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 413, in wait
2012-09-27 12:49:37 TRACE nova _launcher.wait()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 131, in wait
2012-09-27 12:49:37 TRACE nova service.wait()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 166, in wait
2012-09-27 12:49:37 TRACE nova return self._exit_event.wait()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2012-09-27 12:49:37 TRACE nova return hubs.get_hub().switch()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2012-09-27 12:49:37 TRACE nova return self.greenlet.switch()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 192, in main
2012-09-27 12:49:37 TRACE nova result = function(*args, **kwargs)
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 101, in run_server
2012-09-27 12:49:37 TRACE nova server.start()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 162, in start
2012-09-27 12:49:37 TRACE nova self.manager.init_host()
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 269, in init_host
2012-09-27 12:49:37 TRACE nova block_device_info)
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-09-27 12:49:37 TRACE nova return f(*args, **kw)
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 871, in resume_state_on_host_boot
2012-09-27 12:49:37 TRACE nova block_device_info=block_device_info)
2012-09-27 12:49:37 TRACE nova File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 809, in _hard_reboot
2012-09-27 12:49:37 TRACE nova virt_dom = self._conn.lookupByName(instance['name'])
2012-09-27 12:49:37 TRACE nova File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2676, in lookupByName
2012-09-27 12:49:37 TRACE nova if ret is None:raise libvirtError('virDomainLookupByName() failed', conn=self)
2012-09-27 12:49:37 TRACE nova libvirtError: Domain not found: no domain with matching name 'i-00000077'
2012-09-27 12:49:37 TRACE nova

[Jaroslav Pulchart (jaroslav-pulchart-4)]

Looks like a issue https://bugs.launchpad.net/nova/+bug/1026029. I will test proposed patch.

[Jaroslav Pulchart (jaroslav-pulchart-4)]

Yes, the slightly patch from helps 1026029.. But when the host is rebooted when the instance is in between "started->stopped" state if fail again :(

I expect some problem this condition rule
...
            expect_running = (db_state == power_state.RUNNING and
                              drv_state != db_state)
...
in "./compute/manager.py" file "def init_host(self):" function, line 243 (?)

[Jaroslav Pulchart (jaroslav-pulchart-4)]

s/slightly patch/slightly modified patch/

[Jaroslav Pulchart (jaroslav-pulchart-4)]

Arrrrgggg, so many mistakes in one sentence, that is terrible .... I should slow down ...

Fix:
Yes, the slightly modified patch from 1026029 helps. But when the host is rebooted and the instance is in some "between state" ("started->stopped") it fail again :(

[Jason Dillaman (jdillaman)]

What is the failure that you witnessed in comment #2? Is it the same libvirt error from the original problem description or is it just the fact that your VMs are stuck with a task state of powering off?

[Joe Gordon (jogo)]

Is this bug still valid?