[OSSA 2014-017] nova rescue doesn't put VM into RESCUE status on vmware (CVE-2014-2573)

[Jaroslav Henner]

nova rescue of VM on vmWare will create a additional VM ($ORIGINAL_ID-rescue), but after that, the original VM has status ACTIVE. This leads to

[root@jhenner-node ~(keystone_admin)]# nova unrescue foo
ERROR: Cannot 'unrescue' while instance is in vm_state stopped (HTTP 409) (Request-ID: req-792cabb2-2102-47c5-9b15-96c74a9a4819)

the original can be deleted, which then causes leaking of the -rescue VM.

[Maithem (maithem)]

This doesn't seem to be a VC driver issue, I have tried this with the libvirt driver and the same problem happens. The state of the VM is not being set correctly.

[Jaroslav Henner (jhenner)]

https://bugzilla.redhat.com/show_bug.cgi?id=1055531

[Thierry Carrez (ttx)]

Looking into potential security implications.

[Thierry Carrez (ttx)]

I'm not convinced the leaking of the rescue VM is a security issue. Does it let the user circumvent its quota and the extra VMs could be used to overload compute nodes without generating any cost ?

[John Garbutt (johngarbutt)]

This seems specific to vmware, adding vmware tag.

[Thierry Carrez (ttx)]

@John: comment #1 says libvirt is also affected ?

[Tracy Jones (tjones-i)]

@John - that's why i removed it because of comment #1

[Jaroslav Henner (jhenner)]

I have checked that this bug allows the user to pass the limits, so this can lead to DOS and also may lead to consuming more resources than has been paid for.

for i in {1..10}; do
  nova boot --image cirros-0.3.1-x86_64-disk.vmdk --flavor m1.micro foo-$i;
done

# Wait for them to spin-up to ACTIVE state. The limit is 10VMs. No more VMs can be spun up by thi tenant.

for i in {1..10}; do
  nova rescue foo-$i;
done

for i in {1..10}; do
  nova delete foo-$i;
done

# *-rescue machines are still there, in running state
# -- consuming RAM and disk space
# Goto the begining. We can spin-up 10 more VMs.

I have also checked that this problem is not present when using the libvirt/KVM VMs on local disk. The -rescue disk-file is created in the directory of the VM, which gets removed when the VM is being deleted. That holds for Grizzly and Havana release.

[Grant Murphy (gmurphy)]

If Jaroslav's analysis is correct than I think we should move to get a CVE assigned for this. (ttx do you agree?).

I've drafted an impact analysis (below). I'm not sure about the versions. I'm assuming this was introduced by this commit: 8db2f23a43cf8aff21d7ef07af742383c275dd76, however I will need a developer to confirm this problem and the cause.

------

Title: Nova VMWare driver leaks rescued images
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Versions: 2013.2

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova. By
requesting Nova place an image into rescue, then deleting
the image an authenticated user my exceed their quota. This can
result in a denial of service via excessive resource consumption.
Only setups using the Nova VMWare driver are affected.

[Gary Kotton (garyk)]

Correct, the problem is specific to VMware. More specifically if a instance ins deleted after a resuce then a rescus VM will still exist on the backend.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/80284

[Thierry Carrez (ttx)]

Impact description looks good. I would add a comma between "the image" and "an authenticated user", for clarity.
Also would be good to get confirmation on the affected versions.

[Jeremy Stanley (fungi)]

Grant, looks like it should be "Versions: from 2013.2 to 2013.2.2" (assuming it does not affect releases prior to Havana and will be fixed in the 2013.2.3 stable branch point release).

[Grant Murphy (gmurphy)]

@garyk Should this change also be backported to Havana?

[Thierry Carrez (ttx)]

We'll need an havana backport, but maybe wait for the patch to make it to master first to avoid duplication of work

[Grant Murphy (gmurphy)]

This seems to be having some difficulties getting through the gate. Can we please get some more eyes on this?

[Thierry Carrez (ttx)]

master patch blocked on vmware CI. Will need to be backported to milestone-proposed and stable/havana

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/80284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=efb66531bc37ee416778a70d46c657608ca767af
Submitter: Jenkins
Branch: master

commit efb66531bc37ee416778a70d46c657608ca767af
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf
    Closes-bug: 1269418

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/89217

[Thierry Carrez (ttx)]

Anyone up for a stable/havana backport ?

[Nanuk Krinner (nkrinner)]

I'm working on a backport of the original patch written by Gary Kotton , currently running tests.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/90696

[Grant Murphy (gmurphy)]

Backport reviews are in progress here -

https://review.openstack.org/#/c/89762/
https://review.openstack.org/#/c/89768/

Proposed fix above for stable/havana abandoned.

[Thierry Carrez (ttx)]

Above links are for stable/havana.
For referenc, here are the icehouse ones:
https://review.openstack.org/#/c/88514/ (now outdated)
https://review.openstack.org/#/c/89217/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/89217
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ffcb17678c7e5409a1f12a09945b18e8879a677d
Submitter: Jenkins
Branch: stable/icehouse

commit ffcb17678c7e5409a1f12a09945b18e8879a677d
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Closes-bug: 1269418
    (cherry picked from commit efb66531bc37ee416778a70d46c657608ca767af)

    Conflicts:

     nova/virt/vmwareapi/vmops.py

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/89768
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3cc3f62a60662e5bb82136c0cfa464592a6afe9
Submitter: Jenkins
Branch: stable/havana

commit b3cc3f62a60662e5bb82136c0cfa464592a6afe9
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Closes-bug: 1269418
    (cherry picked from commit efb66531bc37ee416778a70d46c657608ca767af)

    Conflicts:

     nova/tests/virt/vmwareapi/test_vmwareapi.py
     nova/virt/vmwareapi/vmops.py

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf

[Jaroslav Henner (jhenner)]

It fails to unrescue when the vmware VM is in powered-on state and also it seems to be leaking when the vm is in ERROR state.

# nova boot --image cirros-0.3.1-x86_64-disk.vmdk --flavor m1.tiny foo
# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ACTIVE | - | Running | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
# nova rescue foo
[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ACTIVE | rescuing | Running | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+

******* Wait until the -rescue VM gets powered on, then the nova seems to switch the state:

[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | RESCUE | - | Shutdown | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
# nova unrescue foo
# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | RESCUE | unrescuing | Shutdown | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ERROR | - | Shutdown | novanetwork=192.168....

[Grant Murphy (gmurphy)]

This issue has been closed. Any incomplete fixes / regressions should be raised in a new (private) bug referencing this one.