[SRU] disk over-commit still not correctly calculated during live migration

Bug #1744079 reported by Matthew Booth on 2018-01-18
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
int32bit
Queens
Low
Lee Yarwood
Rocky
Low
Lee Yarwood
Ubuntu Cloud Archive
High
Unassigned
Queens
High
Unassigned
Rocky
High
Unassigned
nova (Ubuntu)
Status tracked in Disco
Bionic
High
Unassigned
Cosmic
High
Unassigned
Disco
High
Unassigned

Bug Description

[Impact]
nova compares disk space with disk_available_least field, which is possible to be negative, due to overcommit.

So the migration may fail because of a "Migration pre-check error: Unable to migrate dfcd087a-5dff-439d-8875-2f702f081539: Disk of instance is too large(available on destination host:-3221225472 < need:22806528)" when trying a migration to another compute that has plenty of free space in his disk.

[Test Case]
Deploy openstack environment. Make sure there is a negative disk_available_least and a adequate free_disk_gb in one test compute node, then migrate a VM to it with disk-overcommit (openstack server migrate --live <TEST-COMPUTE-NODE> --block-migration --disk-overcommit <VM-NAME>). You will see above migration pre-check error.

This is the formula to compute disk_available_least and free_disk_gb.

disk_free_gb = disk_info_dict['free']
disk_over_committed = self._get_disk_over_committed_size_total()
available_least = disk_free_gb * units.Gi - disk_over_committed
data['disk_available_least'] = available_least / units.Gi

The following command can be used to query the value of disk_available_least

nova hypervisor-show <ID> |grep disk

Steps to Reproduce:
1. set disk_allocation_ratio config option > 1.0
2. qemu-img resize cirros-0.3.0-x86_64-disk.img +40G
3. glance image-create --disk-format qcow2 ...
4. boot VMs based on resized image
5. we see disk_available_least becomes negative

[Regression Potential]
Minimal - we're just changing from the following line:

disk_available_gb = dst_compute_info['disk_available_least']

to the following codes:

if disk_over_commit:
    disk_available_gb = dst_compute_info['free_disk_gb']
else:
    disk_available_gb = dst_compute_info['disk_available_least']

When enabling overcommit, disk_available_least is possible to be negative, so we should use free_disk_gb instead of it by backporting the following two fixes.

https://git.openstack.org/cgit/openstack/nova/commit/?id=e097c001c8e11110efe8879da57264fcb7bdfdf2
https://git.openstack.org/cgit/openstack/nova/commit/?id=e2cc275063658b23ed88824100919a6dfccb760d

This is the code path for check_can_live_migrate_destination:

_migrate_live(os-migrateLive API, migrate_server.py) -> migrate_server -> _live_migrate -> _build_live_migrate_task -> _call_livem_checks_on_host -> check_can_live_migrate_destination

BTW, redhat also has a same bug - https://bugzilla.redhat.com/show_bug.cgi?id=1477706

[Original Bug Report]
Change I8a705114d47384fcd00955d4a4f204072fed57c2 (written by me... sigh) addressed a bug which prevented live migration to a target host with overcommitted disk when made with microversion <2.25. It achieved this, but the fix is still not correct. We now do:

        if disk_over_commit:
            disk_available_gb = dst_compute_info['local_gb']

Unfortunately local_gb is *total* disk, not available disk. We actually want free_disk_gb. Fun fact: due to the way we calculate this for filesystems, without taking into account reserved space, this can also be negative.

The test we're currently running is: could we fit this guest's allocated disks on the target if the target disk was empty. This is at least better than it was before, as we don't spuriously fail early. In fact, we're effectively disabling a test which is disabled for microversion >=2.25 anyway. IOW we should fix it, but it's probably not a high priority.

int32bit (int32bit) on 2018-01-22
Changed in nova:
assignee: nobody → int32bit (int32bit)

Fix proposed to branch: master
Review: https://review.openstack.org/536351

Changed in nova:
status: New → In Progress
Matt Riedemann (mriedem) on 2018-02-05
tags: added: libvirt

Reviewed: https://review.openstack.org/536351
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e2cc275063658b23ed88824100919a6dfccb760d
Submitter: Zuul
Branch: master

commit e2cc275063658b23ed88824100919a6dfccb760d
Author: int32bit <email address hidden>
Date: Mon Jan 22 20:15:21 2018 +0800

    Refix disk size during live migration with disk over-commit

    Currently available disk of targer host is calculated based on
    local_gb, not available disk(free_disk_gb). This check can be
    negative if the target host has no free disk.

    Change-Id: Iec50269ef31dfe090f0cd4db95a37909661bd910
    closes-bug: 1744079

Changed in nova:
status: In Progress → Fix Released
Hua Zhang (zhhuabj) on 2018-10-24
summary: - disk over-commit still not correctly calculated during live migration
+ [SRU] disk over-commit still not correctly calculated during live
+ migration
description: updated
Hua Zhang (zhhuabj) on 2018-10-24
tags: added: sts
Hua Zhang (zhhuabj) on 2018-10-24
tags: added: sts-sru
removed: sts
Hua Zhang (zhhuabj) wrote :
Hua Zhang (zhhuabj) on 2018-10-24
tags: added: sts-sru-needed
removed: sts-sru
Hua Zhang (zhhuabj) on 2018-10-24
tags: added: sts-sponsor
Hua Zhang (zhhuabj) on 2018-10-24
no longer affects: nova (Ubuntu)
Hua Zhang (zhhuabj) on 2018-10-25
tags: added: sts
Matt Riedemann (mriedem) on 2018-10-31
Changed in nova:
importance: Undecided → Low
Changed in nova (Ubuntu Bionic):
status: New → Triaged
Changed in nova (Ubuntu Cosmic):
status: New → Triaged
Changed in nova (Ubuntu Disco):
status: New → Triaged
importance: Undecided → High
Changed in nova (Ubuntu Cosmic):
importance: Undecided → High
Changed in nova (Ubuntu Bionic):
importance: Undecided → High

The attachment "lp1744079_xenial.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Corey Bryant (corey.bryant) wrote :

I've uploaded new versions of nova with the fix for this bug to the disco, cosmic, and bionic unapproved queues.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:18.0.1-0ubuntu3

---------------
nova (2:18.0.1-0ubuntu3) disco; urgency=medium

  * d/tests/nova-shebangs-py3: Update shebang for Python 3.7.

 -- Corey Bryant <email address hidden> Wed, 07 Nov 2018 09:11:09 -0500

Changed in nova (Ubuntu Disco):
status: Triaged → Fix Released

Hello Matthew, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.0.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in nova (Ubuntu Cosmic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in nova (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Matthew, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.6-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-archive:
status: Triaged → Fix Committed
Corey Bryant (corey.bryant) wrote :

Hello Matthew, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
Hua Zhang (zhhuabj) wrote :

Successfully verified queens-proposed nova 2:17.0.6-0ubuntu2~cloud0

tags: added: verification-queens-done
removed: verification-queens-needed
Corey Bryant (corey.bryant) wrote :

Hi Hua, would you also be able to verify cosmic-proposed and bionic-proposed? Those will need to be verified before queens-proposed can be promoted. Thanks.

Corey Bryant (corey.bryant) wrote :

This hasn't been tested yet on cosmic-proposed and we have a stable point release of nova to get out so we're going to have to combine this fix with the stable point release. That means this is going to have to live in cosmic-proposed (and rocky-proposed) for another 7 days minimum.

Brian Murray (brian-murray) wrote :

Hello Matthew, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.0.3-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

James Page (james-page) wrote :

Hello Matthew, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-rocky-needed
Corey Bryant (corey.bryant) wrote :

This hasn't been tested yet on bionic-proposed and we have a stable point release of nova to get out so we're going to have to combine this fix with the stable point release. That means this is going to have to live in bionic-proposed (and queens-proposed) for another 7 days minimum.

Reviewed: https://review.openstack.org/602477
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1c1c373291a93ad6304ff03ea09fea24a5cb5fd0
Submitter: Zuul
Branch: stable/rocky

commit 1c1c373291a93ad6304ff03ea09fea24a5cb5fd0
Author: int32bit <email address hidden>
Date: Mon Jan 22 20:15:21 2018 +0800

    Refix disk size during live migration with disk over-commit

    Currently available disk of targer host is calculated based on
    local_gb, not available disk(free_disk_gb). This check can be
    negative if the target host has no free disk.

    Change-Id: Iec50269ef31dfe090f0cd4db95a37909661bd910
    closes-bug: 1744079
    (cherry picked from commit e2cc275063658b23ed88824100919a6dfccb760d)

Hello Matthew, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.7-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Corey Bryant (corey.bryant) wrote :

Hello Matthew, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
removed: verification-queens-done
Hua Zhang (zhhuabj) wrote :

Hi Corey, sorry for the late reply because I am just back from my pto today, I will verify them ASAP.

Hua Zhang (zhhuabj) wrote :

Successfully verified cosmic-proposed nova 2:18.0.3-0ubuntu1

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Hua Zhang (zhhuabj) wrote :

Successfully verified bionic-proposed nova 2:17.0.7-0ubuntu1

tags: added: verification-done-bionic
removed: verification-needed-bionic
Hua Zhang (zhhuabj) wrote :

Successfully verified queens-proposed nova 2:17.0.7-0ubuntu1~cloud0

tags: added: verification-queens-done
removed: verification-queens-needed
Hua Zhang (zhhuabj) wrote :

Successfully verified rocky-proposed nova 2:18.0.3-0ubuntu1~cloud0

tags: added: verification-rocky-done
removed: verification-rocky-needed
Hua Zhang (zhhuabj) wrote :

Hi Corey, could you pls nominate this fix to xenial as well? we have a customer using xenial-mitaka, I have uploaded a debdiff 'lp1744079_xenial.debdiff', many thanks.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers