VHD snapshot from Hyper-V driver is bigger than original instance

[Matt Riedemann]

This is discussed a bit in bug 1177820. This is on stable/grizzly where the compute node is running the hyper-v driver.

The user has this RHEL 6.4 x86_64 instance:

IBM-SCE> nova show 83604044-a5e0-44ff-b214-d1a91d0ec244
+-------------------------------------+-------------------------------------------------------------+
| Property | Value |
+-------------------------------------+-------------------------------------------------------------+
| status | SHUTOFF |
| updated | 2013-05-08T07:17:53Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | abba-n04-hv01 |
| key_name | None |
| image | RHEL64-Server-x86_64 (f526d15d-5ab8-4217-a450-a1ccb78d984d) |
| VLAN_160 network | 10.160.0.11 |
| hostId | 169746d480604c1349d8c0c0a4ba866211f3f00a2eebf30c87b7390d |
| OS-EXT-STS:vm_state | stopped |
| OS-EXT-SRV-ATTR:instance_name | instance-0000003d |
| OS-EXT-SRV-ATTR:hypervisor_hostname | abba-n04-hv01 |
| flavor | m1.small (2) |
| id | 83604044-a5e0-44ff-b214-d1a91d0ec244 |
| security_groups | [{u'name': u'default'}] |
| user_id | 8e4c4785db6245e7add265e6cf62c67f |
| name | RHELBR-1 |
| created | 2013-05-08T06:56:52Z |
| tenant_id | 6f858c1594b8461cb7000829e3977e9d |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | true |
+-------------------------------------+-------------------------------------------------------------+

With this flavor:

IBM-SCE> nova flavor-show m1.small
+----------------------------+----------+
| Property | Value |
+----------------------------+----------+
| name | m1.small |
| ram | 2048 |
| OS-FLV-DISABLED:disabled | False |
| vcpus | 1 |
| extra_specs | {} |
| swap | |
| os-flavor-access:is_public | True |
| rxtx_factor | 1.0 |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 20 |
| id | 2 |
+----------------------------+----------+

They create a snapshot image of that instance and try to boot an instance from the snapshot but it fails with an ImageTooLarge error. It turns out that the snapshot image is slightly bigger than the original instance. This is the snapshot image details:

Id: 71410397-ba95-4a44-92cd-e62b3bbe1a14
Public: No
Protected: No
Name: RHELBR-B1_backup
Status: active
Size: 21474836992
Disk format: vhd
Container format: bare
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Owner: 6f858c1594b8461cb7000829e3977e9d

When converting that size to GB, it comes to 20.000000477 GB. I'm not sure if they should be using a raw or qcow2 disk format, but it still seems like a problem that the snapshot is reported as bigger than the original instance.

[Alessandro Pilotti (alexpilotti)]

Please: post command outputs and logs in a separate pastebin and not in the bug itself (except short messages of course).

Understanding the issue is very hard otherwise!

Thanks

[Alessandro Pilotti (alexpilotti)]

Can you please post the output of the following Powershell commands execute on teh Hyper-V host?

Get-VHD <path of the original instance VHD>
Get-VHD <path of the snapshot VHD>

Thanks

[HelenLiu (isisliu)]

Reproduce the issue and found that this is not a snapshot issue. As shown in the attachment picture, the base image size is more than 8G, use fixed VHD type and m1.small flavor which has 20G disk, the deployed instance size comes to 20.000000477 GB which bigger than the disk size in flavor. Then take a snapshot of the instance, the snapshot image is 20.000000477 GB, then try to deploy an instance using the snapshot image and m1.small flavor, it fails with an ImageTooLarge error since the snapshot image is bigger than the disk size in flavor.

Question is why use fixed VHD type and m1.small flavor which has 20G disk, the deployed instance size comes to 20.000000477 GB which bigger than the disk size in flavor?

[GuoHui Liu (guohliu)]

I tried with Powershell command to extend the vhd to 20G with below script:
#get the current computer name
$currentname=get-wmiobject Win32_ComputerSystem

$server=$currentname.name

#get a VHD
$disk = Get-WmiObject Msvm_ImageManagementService -namespace "root\virtualization" -computername $server
$path="D:\Hyper-V\instances\_base\test.vhd"

$disk.ExpandVirtualHardDisk($path, 21474836480)

And it seems that the vhd size is still 21474836992 not 21474836480,

[Alessandro Pilotti (alexpilotti)]

Hi, based on your Powershell output, the VHD size is correct. The file size does not matter.

Can you please post a paste bin with the exception trace?

Thanks

[HelenLiu (isisliu)]

Add exception trace on http://paste.openstack.org/show/37384/

[Alessandro Pilotti (alexpilotti)]

The error is due to the fact that the image check is based on the size reported in the metadata which is currently set to the file size:

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1138

To solve the issue, the "size" metadata property should be passed when updating the image in:

https://github.com/openstack/nova/blob/master/nova/virt/hyperv/snapshotops.py#L45

Note: I have to check that the value passed is not getting overridden by the image file size.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29514

[Alessandro Pilotti (alexpilotti)]

Hi, I sent a WIP patch, can you please test it / review it and let me know if it works for you?

[GuoHui Liu (guohliu)]

Hi Alessandro, we are verifying the fix, once get confirm, will let you know asap, thanks for your fix.

[HelenLiu (isisliu)]

Hi Alessandro, We tested after apply the patch at https://review.openstack.org/29514, 2 issues:
1) image_size = instance['root_gb'] * 1204 ** 3, maybe you mistyped 1024 as 1204.
2) even use " image_size = instance['root_gb'] * 1204 ** 3", we took the snapshot which saved to glance using " self._save_glance_image(context, name, image_vhd_path, image_size)", the snapshot image size is still 21474836992, not the "image_size" specified.

[Alessandro Pilotti (alexpilotti)]

If we cannot specify a custom value in Glance for the image size this leaves IMO only one feasible option: Including the header size during the VHD resize operation on boot.

I'm bringing this today to the Hyper-V meeting.

[HelenLiu (isisliu)]

Hi Alessandro, this is Xiao Pei. I think what we have discussed at the Hyper-V meeting is good to me to solve the problem. After you deliver a new patch, I can verify it in my environment for deploy instance, capture instance and resize instance etc. If there is anything I can do to help move the review along to approval, feel free to let me know. Thanks.

[Alessandro Pilotti (alexpilotti)]

One additional note: since we are in the process of adding VHDX support, we need to consider this as well in a separate patch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31228
Committed: http://github.com/openstack/nova/commit/690cf0bdbe57bc05aaf013ab39e050d04a0cd9b7
Submitter: Jenkins
Branch: master

commit 690cf0bdbe57bc05aaf013ab39e050d04a0cd9b7
Author: guohliu <email address hidden>
Date: Wed Jun 5 15:37:01 2013 +0800

    Fix hyper-v vhd real size bigger than flavor issue

    In hyper-v, after deploy the image, real vhd file size will be
    extended from flavor size plus vhd metadata size, this patch
    fix this issue by check the vhd type and minus the vhd metadata
    size according to the specific vhd type.

    Fixes bug #1177927

    Change-Id: I51223f4f778f33e73dce1e0f53e23e8d7ac77904

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/47080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47080
Committed: http://github.com/openstack/nova/commit/00e022afaecd7a691156bd51efeb7eaf650aac2d
Submitter: Jenkins
Branch: stable/grizzly

commit 00e022afaecd7a691156bd51efeb7eaf650aac2d
Author: guohliu <email address hidden>
Date: Wed Jun 5 15:37:01 2013 +0800

    Fix hyper-v vhd real size bigger than flavor issue

    In hyper-v, after deploy the image, real vhd file size will be
    extended from flavor size plus vhd metadata size, this patch
    fix this issue by check the vhd type and minus the vhd metadata
    size according to the specific vhd type.

    Fixes bug #1177927

    Conflicts:
            nova/tests/test_hypervapi.py
            nova/tests/virt/hyperv/test_vhdutils.py
            nova/virt/hyperv/constants.py

    Change-Id: I51223f4f778f33e73dce1e0f53e23e8d7ac77904
    (cherry picked from commit 690cf0bdbe57bc05aaf013ab39e050d04a0cd9b7)