Ubuntu 22.04 Local Volume Live Migration Issue

[Ammad Ali]

Hi,

Using ubuntu 22.04 LTS and setup openstack yoga on it. The live migration of cinder backed volumes are successful but the migration of local volume (i.e root disks resides on local compute storage) goes fail. However the migration works fine on ubuntu 20.04 yoga release from UCA with qemu 4.2 and libvirt 8.0. Issue looks to be in below qemu version.

Qemu 6.2 (default in ubuntu 22.04)
Libvirt 8.0 (default in ubuntu 22.04)

The migration of local storage instance are getting failed with below error in nova-compute logs. I think the error is being thrown from libvirt.

2022-09-13 11:02:17.047 3439 ERROR nova.compute.manager [-] [instance: 113ed3d7-b371-45d7-aeac-90c7cae4e2a4] Post live migration at destination kvm10-a1-khi01 failed: nova.exception_Remote.InstanceNotFound_Remote: Instance 113ed3d7-b371-45d7-aeac-90c7cae4e2a4 could not be found.
Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/nova/virt/libvirt/host.py", line 686, in _get_domain
    return conn.lookupByUUIDString(instance.uuid)

  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
   result = proxy_call(self._autowrap, f, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
    rv = execute(f, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
    six.reraise(c, e, tb)

  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value

  File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
    rv = meth(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/libvirt.py", line 4920, in lookupByUUIDString
    raise libvirtError('virDomainLookupByUUIDString() failed')

libvirt.libvirtError: Domain not found: no domain with matching uuid '113ed3d7-b371-45d7-aeac-90c7cae4e2a4'

[Ammad Ali (syedammad83)]

This looks like a bug in QEMU. The bug is be tracked here:

https://bugzilla.redhat.com/show_bug.cgi?id=1990835
https://bugzilla.redhat.com/show_bug.cgi?id=2033626

The bug is reportedly fixed in this commit:

https://github.com/qemu/qemu/commit/e15f3a66c830e3fce99c9d56c493c2f7078a1225

The commit however is part of version 7.0.0 or newer. Is it possible to backport above commit in qemu 6.2 in ubuntu 22.04 ?

[Sergio Durigan Junior (sergiodj)]

Thank you for taking the time to report a bug.

Would you happen to have a corefile/backtrace log for qemu? The traceback excerpt you posted belongs to libvirt, but I could not find anything related to qemu specifically. I could not establish a direct relation between the problem you reported and the bugs you linked, other than the fact that one of the bugs talks about issues when performing storage migrations.

In order to be able to backport the fix to Jammy's qemu, we need to be able to reproduce the issue and confirm that it is indeed what you're seeing in your environment.

Thanks!

[Sergio Durigan Junior (sergiodj)]

I'm setting this bug as Incomplete for now. Please set it back to New once you've provided the requested info. Thanks