Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'

[Satish Patel]

I have running openstack pike 16.0.15 and seeing following error when trying to do live migration with ceph storage.

Any idea what is the workaround here?

2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [req-8049c16f-0a42-4c15-8806-346d0ea556b7 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] Traceback (most recent call last):
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5580, in _do_live_migration
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] block_migration, migrate_data)
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6436, in live_migration
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] migrate_data)
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6944, in _live_migration
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] guest.migrate_configure_max_speed(
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948]

[Jordan Callicoat (jcallicoat)]

The error is from libvirt driver calling `guest.migrate_configure_max_speed` here:

https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6944

That method was not added to the guest module until after Pike (https://github.com/openstack/nova/commit/23446a9552b5be3b040278646149a0f481d0a005) and it doesn't appear to have been backported to Pike as it is missing here:

https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/guest.py

The call to `guest.migrate_configure_max_speed` was added to the driver in this commit: https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348

[Satish Patel (satish-txt)]

After adding following stanza in guest.py file it works, Thanks you Jorden!

def migrate_configure_max_speed(self, bandwidth):
        """The maximum bandwidth that will be used to do migration
        :param bw: Bandwidth in MiB/s
        """
        self._domain.migrateSetMaxSpeed(bandwidth)

[Satish Patel (satish-txt)]

Why this beautiful fix not backported to pike?

[Matt Riedemann (mriedem)]

There is a patch here: https://review.openstack.org/#/c/586965/

[Satish Patel (satish-txt)]

Thanks Matt,

Finally i migrated over queens which has this patch already.

I would be good if you guys promote this patch to pike also

On Mon, Jul 30, 2018 at 10:55 AM, Matt Riedemann <email address hidden> wrote:
> There is a patch here: https://review.openstack.org/#/c/586965/
>
> ** Also affects: nova/pike
> Importance: Undecided
> Status: New
>
> ** Changed in: nova
> Status: New => Invalid
>
> ** Changed in: nova/pike
> Status: New => In Progress
>
> ** Changed in: nova/pike
> Importance: Undecided => High
>
> ** Changed in: nova/pike
> Assignee: (unassigned) => Lee Yarwood (lyarwood)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1783635
>
> Title:
> Live migration failed.: AttributeError: 'Guest' object has no
> attribute 'migrate_configure_max_speed'
>
> Status in OpenStack Compute (nova):
> Invalid
> Status in OpenStack Compute (nova) pike series:
> In Progress
>
> Bug description:
> I have running openstack pike 16.0.15 and seeing following error when
> trying to do live migration with ceph storage.
>
> Any idea what is the workaround here?
>
>
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [req-8049c16f-0a42-4c15-8806-346d0ea556b7 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] Traceback (most recent call last):
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5580, in _do_live_migration
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] block_migration, migrate_data)
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6436, in live_migration
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] migrate_data)
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6944, in _live_migration
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] guest.migrate_configure_max_speed(
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948] AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
> 2018-07-25 16:14:15.180 59135 ERROR nova.compute.manager [instance: 3971f7e6-33a4-47f8-a308-39dff169f948]
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/178...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/586965
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e0f1c2cbfd46c0dad72a7f0fb0a1c7fb91fd0c6b
Submitter: Zuul
Branch: stable/pike

commit e0f1c2cbfd46c0dad72a7f0fb0a1c7fb91fd0c6b
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Thu Aug 24 09:47:29 2017 -0400

    libvirt: add method to configure migration speed

    In this commit we are enhancing guest object to control the maximum bw
    to perform migration.

    Closes-Bug: #1783635
    Related-Bug: #1414559
    Change-Id: I35470773b8c467449ed71217fdb4b6c82f455e33
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    (cherry picked from commit 23446a9552b5be3b040278646149a0f481d0a005)

[Marek Grudzinski (ivve)]

Hi, is this depending on any other changes? I have a pike installation but all migrations are stuck at 1024kb/sec per host on outgoing migrations. Incoming migrations are not capped.
This is after changing the suggested stanza.

Tried changing nova.conf @ two of the test compute hosts and restarted nova-compute to no avail.

live_migration_bandwidth = 1024

also tried values 4096 and 0

always stuck @ 1024KB/sec through the entire migration (testing a 2gb ram cirros instance). It never speeds up.

Versions for nova & neutron are the following:
neutron & its agents 11.0.6.dev47
nova 16.1.5

Also using ceph & linuxbridges.

Also can't seem to migrate between hosts with different cpu model although both intel.
Intel(R) Xeon(R) Platinum 8176
Intel(R) Xeon(R) CPU E5-2680 v4

[Satish Patel (satish-txt)]

@Marek,

Look like you are missing this patch: https://bugs.launchpad.net/nova/+bug/1786346

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.