Unable to live-migrate : Disk of instance is too large

Bug #1708572 reported by lanrui280 on 2017-08-04
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Matthew Booth
Ubuntu Cloud Archive
High
Unassigned
Mitaka
High
Unassigned
Ocata
High
Unassigned
Pike
High
Unassigned
Queens
High
Unassigned
nova (Ubuntu)
High
Unassigned
Xenial
High
Unassigned
Bionic
High
Unassigned

Bug Description

os:centos7.3
openstack:ocata

when I tried to live-migrate an instance,it is wrong:
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 686, in decorated_function
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5281, in check_can_live_migrate_source
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server block_device_info)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5750, in check_can_live_migrate_source
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server block_device_info)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5874, in _assert_dest_node_has_enough_disk
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=reason)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server MigrationPreCheckError: Migration pre-check error: Unable to migrate e9411b18-ce84-4094-8811-ad9675c245f3: Disk of instance is too large(available on destination host:-458487758848 < need:21692416)
2017-07-28 11:33:03.917 18473 ERROR oslo_messaging.rpc.server.

I found the free space of the destination host is -458487758848(A negative number).
The same bug is in https://bugs.launchpad.net/mos/+bug/1650490
But I don't kwon how to update my nova.

-------------------------------------------------------------------

=== Ubuntu SRU Details ===

[Impact]
See above and see https://bugs.launchpad.net/nova/+bug/1744079.

[Test Case]
See https://bugs.launchpad.net/nova/+bug/1744079.

[Regression Potential]
See https://bugs.launchpad.net/nova/+bug/1744079.

no longer affects: mos
tags: added: libvirt live-migration openstack-version.ocata
Sylvain Bauza (sylvain-bauza) wrote :

Are you using OpenStackClient for calling the live-migration action ? Which exact CLI call do you do ?

Changed in nova:
status: New → Incomplete
lanrui280 (lanrui456) wrote :

I just used 'live migration' button of dashboard with 'disk over commit' and 'block migration' options.@Sylvain Bauza (sylvain-bauza)

Eli Qiao (taget-9) on 2017-11-22
Changed in nova:
status: Incomplete → New

Fix proposed to branch: master
Review: https://review.openstack.org/524681

Changed in nova:
assignee: nobody → Matthew Booth (mbooth-9)
status: New → In Progress

Reviewed: https://review.openstack.org/524681
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e097c001c8e11110efe8879da57264fcb7bdfdf2
Submitter: Zuul
Branch: master

commit e097c001c8e11110efe8879da57264fcb7bdfdf2
Author: Matthew Booth <email address hidden>
Date: Fri Dec 1 16:39:49 2017 +0000

    Fix disk size during live migration with disk over-commit

    Prior to microversion 2.25, the migration api supported a
    'disk_over_commit' parameter, which indicated that we should do disk
    size check on the destination host taking into account disk
    over-commit. Versions since 2.25 no longer have this parameter, but we
    continue to support the older microversion.

    This disk size check was broken when disk over commit was in use on
    the host. In LibvirtDriver._assert_dest_node_has_enough_disk() we
    correctly calculate the required space using allocated disk size
    rather than virtual disk size when doing an over-committed check.
    However, we were checking against 'disk_available_least' as reported
    by the destination host. This value is: the amount of disk space which
    the host would have if all instances fully allocated their storage. On
    an over-committed host it will therefore be negative, despite there
    actually being space on the destination host.

    The check we actually want to do for disk over commit is: does the
    target host have enough free space to take the allocated size of this
    instance's disks? We leave checking over-allocation ratios to the
    scheduler. Note that if we use disk_available_least here and the
    destination host is over-allocated, this will always fail because free
    space will be negative, even though we're explicitly ok with that.

    Using disk_available_least would make sense for the non-overcommit
    case, where the test would be: would the target host have enough free
    space for this instance if the instance fully allocated all its
    storage, and everything else on the target also fully allocated its
    storage? As noted, we no longer actually run that test, though.

    We fix the issue for legacy microversions by fixing the destination
    host's reported disk space according to the given disk_over_commit
    parameter.

    Change-Id: I8a705114d47384fcd00955d4a4f204072fed57c2
    Resolves-bug: #1708572

Changed in nova:
status: In Progress → Fix Released
Lee Yarwood (lyarwood) on 2018-01-09
Changed in nova:
importance: Undecided → High
Amit Bhardwaj (bhardwajamit23) wrote :

How to get this patch in Newton?

Fabo Yi (folkart) wrote :

When I launched an instance using a volume stored on IP-SAN, 'Disk of instance is too large' exception was raised, and the amount of disk space available was also a negative number. It seems that all allocated IP-SAN volumes used to create instances are taken into account when computing the free space on the target host.

The fix in https://review.openstack.org/524681 does an over-committed check, but I'm not sure it would help if we launch an instance from a Cinder storage backend.

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/530743
Reason: I'm just going to abandon. If anyone cares, restore and sort this out.

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/530744
Reason: I'm just going to abandon. If anyone cares, restore and sort this out.

Change abandoned by Zhang Hua (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/631372

Reviewed: https://review.openstack.org/530743
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a4debb3dd2726299bbee0c07b7fc052ac72555b
Submitter: Zuul
Branch: stable/pike

commit 1a4debb3dd2726299bbee0c07b7fc052ac72555b
Author: Matthew Booth <email address hidden>
Date: Fri Dec 1 16:39:49 2017 +0000

    Fix disk size during live migration with disk over-commit

    Prior to microversion 2.25, the migration api supported a
    'disk_over_commit' parameter, which indicated that we should do disk
    size check on the destination host taking into account disk
    over-commit. Versions since 2.25 no longer have this parameter, but we
    continue to support the older microversion.

    This disk size check was broken when disk over commit was in use on
    the host. In LibvirtDriver._assert_dest_node_has_enough_disk() we
    correctly calculate the required space using allocated disk size
    rather than virtual disk size when doing an over-committed check.
    However, we were checking against 'disk_available_least' as reported
    by the destination host. This value is: the amount of disk space which
    the host would have if all instances fully allocated their storage. On
    an over-committed host it will therefore be negative, despite there
    actually being space on the destination host.

    The check we actually want to do for disk over commit is: does the
    target host have enough free space to take the allocated size of this
    instance's disks? We leave checking over-allocation ratios to the
    scheduler. Note that if we use disk_available_least here and the
    destination host is over-allocated, this will always fail because free
    space will be negative, even though we're explicitly ok with that.

    Using disk_available_least would make sense for the non-overcommit
    case, where the test would be: would the target host have enough free
    space for this instance if the instance fully allocated all its
    storage, and everything else on the target also fully allocated its
    storage? As noted, we no longer actually run that test, though.

    We fix the issue for legacy microversions by fixing the destination
    host's reported disk space according to the given disk_over_commit
    parameter.

    Change-Id: I8a705114d47384fcd00955d4a4f204072fed57c2
    Resolves-bug: #1708572
    (cherry picked from commit e097c001c8e11110efe8879da57264fcb7bdfdf2)

tags: added: in-stable-pike

Reviewed: https://review.openstack.org/530744
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e2d2c3e8b14a5461c95014be3accb1e40f3d1a0b
Submitter: Zuul
Branch: stable/ocata

commit e2d2c3e8b14a5461c95014be3accb1e40f3d1a0b
Author: Matthew Booth <email address hidden>
Date: Fri Dec 1 16:39:49 2017 +0000

    Fix disk size during live migration with disk over-commit

    Prior to microversion 2.25, the migration api supported a
    'disk_over_commit' parameter, which indicated that we should do disk
    size check on the destination host taking into account disk
    over-commit. Versions since 2.25 no longer have this parameter, but we
    continue to support the older microversion.

    This disk size check was broken when disk over commit was in use on
    the host. In LibvirtDriver._assert_dest_node_has_enough_disk() we
    correctly calculate the required space using allocated disk size
    rather than virtual disk size when doing an over-committed check.
    However, we were checking against 'disk_available_least' as reported
    by the destination host. This value is: the amount of disk space which
    the host would have if all instances fully allocated their storage. On
    an over-committed host it will therefore be negative, despite there
    actually being space on the destination host.

    The check we actually want to do for disk over commit is: does the
    target host have enough free space to take the allocated size of this
    instance's disks? We leave checking over-allocation ratios to the
    scheduler. Note that if we use disk_available_least here and the
    destination host is over-allocated, this will always fail because free
    space will be negative, even though we're explicitly ok with that.

    Using disk_available_least would make sense for the non-overcommit
    case, where the test would be: would the target host have enough free
    space for this instance if the instance fully allocated all its
    storage, and everything else on the target also fully allocated its
    storage? As noted, we no longer actually run that test, though.

    We fix the issue for legacy microversions by fixing the destination
    host's reported disk space according to the given disk_over_commit
    parameter.

    Change-Id: I8a705114d47384fcd00955d4a4f204072fed57c2
    Resolves-bug: #1708572
    (cherry picked from commit e097c001c8e11110efe8879da57264fcb7bdfdf2)

tags: added: in-stable-ocata
Changed in cloud-archive:
status: New → Fix Released
importance: Undecided → High
Changed in nova (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → High
Changed in nova (Ubuntu):
status: New → Fix Released
importance: Undecided → High
Changed in nova (Ubuntu Bionic):
status: New → Fix Released
importance: Undecided → High
status: Fix Released → Invalid
Changed in nova (Ubuntu):
status: Fix Released → Invalid
Changed in cloud-archive:
status: Fix Released → Invalid
Changed in nova (Ubuntu Bionic):
status: Invalid → Fix Released
description: updated
description: updated

Hello lanrui280, or anyone else affected,

Accepted nova into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:13.1.4-0ubuntu4.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Xenial):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-xenial
Corey Bryant (corey.bryant) wrote :

Hello lanrui280, or anyone else affected,

Accepted nova into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-mitaka-needed
Corey Bryant (corey.bryant) wrote :

Tagged xenial and mitaka as verified in line with https://bugs.launchpad.net/nova/+bug/1744079.

tags: added: verification-done-xenial verification-mitaka-done
removed: verification-mitaka-needed verification-needed-xenial

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:13.1.4-0ubuntu4.4

---------------
nova (2:13.1.4-0ubuntu4.4) xenial; urgency=medium

  * Refix disk size during live migration with disk over-commit
    - (LP: #1708572) and (LP: #1744079)
    - d/p/0001-Fix-disk-size-during-live-migration-with-disk-over-c.patch
    - d/p/0002-Refix-disk-size-during-live-migration-with-disk-over.patch

 -- Zhang Hua <email address hidden> Tue, 02 Apr 2019 18:48:16 +0800

Changed in nova (Ubuntu Xenial):
status: Fix Committed → Fix Released
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package nova - 2:13.1.4-0ubuntu4.4~cloud0
---------------

 nova (2:13.1.4-0ubuntu4.4~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:13.1.4-0ubuntu4.4) xenial; urgency=medium
 .
   * Refix disk size during live migration with disk over-commit
     - (LP: #1708572) and (LP: #1744079)
     - d/p/0001-Fix-disk-size-during-live-migration-with-disk-over-c.patch
     - d/p/0002-Refix-disk-size-during-live-migration-with-disk-over.patch

Changed in nova (Ubuntu):
status: Invalid → Fix Released
Changed in cloud-archive:
status: Invalid → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers