[OSSA 2014-037] vmware: deletion VM in resize state will cause VM-orig get leak (CVE-2014-8333)

[zhu zhu]

For vmware vcenter driver, resize a VM, during resizing , at the same time, delete the vm, the VM-orig can not be deleted in ESXi host. So makes VM leaks.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the report, the OSSA task is set to incomplete pending additional security review from nova-coresec.

[Andrew Laski (alaski)]

The VMWare driver renames the instance on the source host to be instance['uuid']-orig during a resize/migration. And it does this before the vm_state goes to RESIZED. This is problematic because

1. The deletion process will confirm any resizes which are in the RESIZED state so they are cleaned up properly. But since there's a window between the renaming of the instance and vm state going to RESIZED it is possible that the resize won't be confirmed first.

2. The driver method for deleting an instance does not check for the presence of an instance named instance['uuid']-orig.

So it is possible to leak these VMs.

[zhu zhu (zhuzhubj)]

yes, andrew, exactly during the window for resize step migration_disk_and_poweroff after disassociate VM(rename) or new VM clone but before migration status to be finished, the deletion of VM will not delete those uuid-orig VMs.

Only when the VM arrive to Resized state(VerifiedResize), the deletion will try to confirm resize to cleanup and also deletion of the orig vms.

[zhu zhu (zhuzhubj)]

https://review.openstack.org/#/c/118595/

[Tristan Cacqueray (tristan-cacqueray)]

Ok so I've confirmed this bug based on alaski's comment #2.

And now that there is a public review so we should problably mark this as Public Security.

@Zhuzhu, next time, if you want the bug to remain private, please attach patch to the LP page...

[Tristan Cacqueray (tristan-cacqueray)]

Here is impact description draft #1:

Title: Nova VMware instance leak potentially leading to compute DoS
Reporter: Zhu Zhu (IBM)
Products: Nova
Versions: up to 2014.1.2

Description:
Zhu Zhu from IBM reported a vulnerability in Nova VMware driver. By deleting an instance while it is in re-size state, an authenticated user may leak the original instance and use this to conduct a denial of service attack on compute nodes. All Nova VMware setups are affected.

[Thierry Carrez (ttx)]

Maybe elaborate on "leak" here -- it doesn't leak out to someone else, it just gets stuck, and those can pile up to a DoS.

[John Garbutt (johngarbutt)]

@ttx - that is my take on this too, looks like a DoS attack by using up capacity without being charged for it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118595
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e1f8664c9fa83f77f5bb763ffcc3157905ed954c
Submitter: Jenkins
Branch: master

commit e1f8664c9fa83f77f5bb763ffcc3157905ed954c
Author: ZHU ZHU <email address hidden>
Date: Wed Sep 3 03:59:13 2014 -0500

    VMWare: Fix VM leak when deletion of VM during resizing

    During the VM resizing, before VM arrive RESIZED state, driver
    migrate_disk_and_power_off will initially rename orginal vm
    'uuid' to be 'uuid-orig' and clone a new vm with 'uuid' name.
    When deletion VM is triggered at this time window, it wouldn't
    be able to delete the VM uuid-orig in VCenter and so cause VM leak.
    As VM task state will be set to 'deleting' and it can not be used to
    determine the resize migrating/migrated state, this fix will
    attempt to delete orig VM within destroy phase.

    Change-Id: I7598afbf0dc3c527471af34224003d28e64daaff
    Closes-Bug: #1359138

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/125492

[Tristan Cacqueray (tristan-cacqueray)]

I've proposed a backport to stable/icehouse here: https://review.openstack.org/125492

Here is impact description draft #2 (reworded after our usual VMware leak impact description...):

Title: Nova VMware instance in re-size state leaks
Reporter: Zhu Zhu (IBM)
Products: Nova
Versions: up to 2014.1.2

Description:
Zhu Zhu from IBM reported a vulnerability in Nova VMware driver. If an authenticated user delete an instance while it is in re-size state, it will cause the original instance to not be deleted. An attacker can use this to launch a denial of service attack. All Nova VMware setups are affected.

[Thierry Carrez (ttx)]

Perhaps:
"Title: Nova VMware instance in re-size state may leak"

otherwise looks good.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks ttx!

Dan Genin made some comments to the stable/icehouse backport ( https://review.openstack.org/#/c/125492 patchset 4)
Could these also apply to the juno/kilo part ?

Title: Nova VMware instance in re-size state may leak
Reporter: Zhu Zhu (IBM)
Products: Nova
Versions: up to 2014.1.2

Description:
Zhu Zhu from IBM reported a vulnerability in Nova VMware driver. If an authenticated user delete an instance while it is in re-size state, it will cause the original instance to not be deleted. An attacker can use this to launch a denial of service attack. All Nova VMware setups are affected.

[Jeremy Stanley (fungi)]

On Tristan's impact description in comment #13, this also affects nova up to 2014.1.3 now right? Also, I see a verb tense mistake--should be: "If an authenticated user deletes an instance..." and "resize" is generally one word (not hyphenated).

[Tristan Cacqueray (tristan-cacqueray)]

Indeed, there is no reason why 2014.1.3 wouldn't be affected, and thanks for the corrections fungi!

Title: Nova VMware instance in resize state may leak
Reporter: Zhu Zhu (IBM)
Products: Nova
Versions: up to 2014.1.3

Description:
Zhu Zhu from IBM reported a vulnerability in Nova VMware driver. If an authenticated user deletes an instance while it is in resize state, it will cause the original instance to not be deleted. An attacker can use this to launch a denial of service attack. All Nova VMware setups are affected.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/125492
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d71445c7d2d2921d10a08f82330f0ab8ef4f7df2
Submitter: Jenkins
Branch: stable/icehouse

commit d71445c7d2d2921d10a08f82330f0ab8ef4f7df2
Author: ZHU ZHU <email address hidden>
Date: Wed Sep 3 03:59:13 2014 -0500

    VMWare: Fix VM leak when deletion of VM during resizing

    During the VM resizing, before VM arrive RESIZED state, driver
    migrate_disk_and_power_off will initially rename orginal vm
    'uuid' to be 'uuid-orig' and clone a new vm with 'uuid' name.
    When deletion VM is triggered at this time window, it wouldn't
    be able to delete the VM uuid-orig in VCenter and so cause VM leak.
    As VM task state will be set to 'deleting' and it can not be used to
    determine the resize migrating/migrated state, this fix will
    attempt to delete orig VM within destroy phase.

    Conflicts:
     nova/tests/virt/vmwareapi/test_driver_api.py
     nova/virt/vmwareapi/vmops.py

    Closes-Bug: #1359138

    NOTE: the aformentioned patch broke Minesweeper. The fix was also
    cherry picked from commit e464bc518e8590d59c2741948466777982ca3319.
    This was to do two things:
    1. Solve the actual bug
    2. Ensure that the unit tests and Minesweeper passed

    Change-Id: I7598afbf0dc3c527471af34224003d28e64daaff
    (cherry-picked from e1f8664c9fa83f77f5bb763ffcc3157905ed954c)

[Thierry Carrez (ttx)]

Impact desc +1