Resized instance doesn't track new flavor, leading to possible instance-from-snapshot creation failures

Bug #1565590 reported by John Wood
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
Medium
Unassigned

Bug Description

To reproduce the issue, follow these steps:
1) Create s small instance
2) Resize to a larger instance
3) Create a snapshot image from the resized instance
4) Create an instance from this snapshot image, selecting the smallest flavor based on the snapshot image’s ‘min_disk’ value

The expected result is that in step #4 the image’s ‘min_disk’ value can be used to select a minimum sized flavor for a new instance, leading to a successful instance creation.

The actual result is that the minimum sized flavor suggested by ‘min_disk’ is too small for the snapshot image, so instance creation fails with a 'VM too large for selected flavor' error here: https://github.com/openstack/nova/blob/master/nova/virt/xenapi/vm_utils.py#L901

This error is seen in a deployment using Xen hypervisors with Glance/Swift for image storage.

For more detailed information:

When an instance is created information about it’s image is placed into the instance’s system metadata. One image property is specially treated however: min_disk. The instance’s ‘min_disk’ value is set based on the re-size flavor’s ‘root_gb’ (see https://github.com/openstack/nova/blob/master/nova/utils.py#L1229). When a snapshot image is created from the instance, the instance’s ‘min_disk’ value is added to image’s properties. This ‘min_disk’ setting can then be used as per step #4 above.

When the instance is resized (typically to a larger flavor) the instance’s ‘min_disk’ is *not* updated with the new flavor’s ‘root_gb’ (however the instance’s ‘root_gb’ is updated from the flavor). Hence when a snapshot image is created from this instance, the resultant image’s ‘min_disk’ is also set to the pre-resized value. For resize ups then, this value is then too small for the virtual disk size the snapshot will consume when instances are created from it.

A proposed fix for this issue then is to set the snapshot image’s ‘min_disk’ property in a similar way to the nova/utils.py call above:

     min_disk = instance’s min_disk
     if flavor:
         if instance’s disk_format == ‘vhd’:
             min_disk = resize flavor’s root_gb
         else:
             min_disk = max(min_disk, resize flavor’s root_gb)

The instance’s ‘min_disk’ could stay set to the original image properties for historical purposes.

Tags: resize xen
John Wood (john-wood-w)
Changed in nova:
assignee: nobody → John Wood (john-wood-w)
tags: added: xen
John Wood (john-wood-w)
description: updated
Matt Riedemann (mriedem)
tags: added: resize
Changed in nova:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/301966

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/301966
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the nova review queue sane. Please feel free to restore the change if you're still working on it.

Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in nova:
status: In Progress → Triaged
assignee: John Wood (john-wood-w) → nobody
Revision history for this message
Spencer Yu (yushb) wrote :

Hi Nova cores, this problem seems still in master branch, could you give some advice on how to resolve it? Thanks.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Is bug 1646740 a duplicate of this?

Revision history for this message
Matt Riedemann (mriedem) wrote :

> Is bug 1646740 a duplicate of this?

Maybe not, sounds like this bug is more about properly updating the instance.image metadata to update the min_disk when resizing the instance to a larger flavor so that when you create a snapshot of the instance and try to boot from that snapshot with a smaller flavor it fails.

Seems we could write a functional test to recreate this?

Revision history for this message
Matt Riedemann (mriedem) wrote :

@Spencer: do you hit this when using the libvirt driver or only the xenapi driver? Because the latter was deprecated in the Train release.

Revision history for this message
Matt Riedemann (mriedem) wrote :

I was able to recreate this with devstack on master using the libvirt driver so it's not xen specific, details are in:

http://paste.openstack.org/show/784729/

I think a simple way to fix this is probably just update the image_min_disk value to the new flavor when confirming a resize.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.