Live migration fails with "Disk of instance is too large" error

[Denis Kostryukov]

MOS 7.0

When we tried to live-migrate an instance, we have got the following error in nova logs:

2016-12-08 17:12:36.557 128792 TRACE oslo_messaging.rpc.dispatcher raise exception.MigrationPreCheckError(reason=reason)
2016-12-08 17:12:36.557 128792 TRACE oslo_messaging.rpc.dispatcher MigrationPreCheckError: Migration pre-check error: Unable to migrate a67f2b0d-03f4-44d5-8258-f9e84dbec03d: Disk of instance is too large(available on destination host:-15032385536 < need:2293760)
2016-12-08 17:12:36.557 128792 TRACE oslo_messaging.rpc.dispatcher
2016-12-08 17:12:36.560 128792 ERROR oslo_messaging._drivers.common [req-feec0c58-4814-44a5-8124-008e4feb1f50 d24897d938d84e6984a3600392038978 a0c52dc4c2854d19aad6de1303ca8753 - - -] Returning exception Migration pre-check error: Unable to migrate a67f2b0d-03f4-44d5-8258-f9e84dbec03d: Disk of instance is too large(available on destination host:-15032385536 < need:2293760) to caller

There is the same bug in Nova project, the issue was fixed in Liberty:
https://bugs.launchpad.net/nova/+bug/1480441

Please provide a backport for MOS 7.0

[Denis Meltsaykin (dmeltsaykin)]

Denis, it seems the bug you are facing is not the same as https://bugs.launchpad.net/nova/+bug/1480441, since you have a negative number of space available which is very strange.

[Denis Kostryukov (dkostryukov)]

Hello Denis.
In https://bugs.launchpad.net/nova/+bug/1480441 I can see also negative value of available disk space:
Disk of instance is too large(available on destination host:-38654705664 < need:1073741824)

But anyway the negative value is very strange.

[Denis Meltsaykin (dmeltsaykin)]

Yeah, but the fix adds a retry to the migration procedure. I'm not sure it would help if you have such a strange value. Anyways we will investigate it someday, if you have a diagnostic snapshot it would be nice too.

[Alexey Stupnikov (astupnikov)]

I have doublechecked nova's code and have the following conclusions:

1. An original error is raised by libvirt driver (virt.libvirt.driver). The function _assert_dest_node_has_enough_disk is used to process RPC call from controller and provide information compute's capability to host migrated instance. NOTE, driver substracts CONF.reserved_host_disk_mb value from actual storage size; it should be a reason of large negative values.
2. Since it is totally possible to make compute generate such exceptions, it will be reasonable to process them correctly. We have to fix this issue.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Chris St. Pierre <email address hidden>
Review: https://review.fuel-infra.org/32292

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32292
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 49f244c65c2ba6040325f9b5e4c36fa0a766c125
Author: Chris St. Pierre <email address hidden>
Date: Thu Mar 16 17:17:11 2017

Retry live migration on pre-check failure

Make Nova continue trying to find a host to live migrate an instance
to when a possible destination host has failed migration pre-checks
with MigrationPreCheckError, which can be raised if a hypervisor disk
is full.

Closes-Bug: #1650490
(cherry-picked from commit 72fb7fdddb675206eca3d2ed19b6901c0e1e6975)
Change-Id: I4fc4141eafbf7665e020c67de9578ef12c1c5ac5

[Ekaterina Shutova (eshutova)]

Verified on 7.0 mu8 updates.
Before updates live migration was failed with error(errors were also in nova log) if disk on one of compute's was full:
root@node-1:~# nova live-migration --block-migrate vm1
ERROR (BadRequest): Migration pre-check error: Unable to migrate b919a4ba-7add-4ac7-b35c-3418872edb41: Disk of instance is too large(available on destination host:0 < need:41126400) (HTTP 400) (Request-ID: req-b8897443-8282-42d2-a249-8e0101ec7922)
After apdates live migration passed well to the compute node with available disk space.

[long (long-cheung)]

Dear all bro,

How to update to new version?

i am face same problem.

>.<

[Alexey Stupnikov (astupnikov)]

Wassup dear @long-cheung! We will release update in the next three weeks, you will get updated package from mos7.0-updates repo. Keep an eye on mirantis documentation here:
https://docs.mirantis.com/openstack/fuel/fuel-7.0/maintenance-updates.html