[OSSA 2014-017] nova rescue doesn't put VM into RESCUE status on vmware (CVE-2014-2573)

Bug #1269418 reported by Jaroslav Henner on 2014-01-15
266
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Gary Kotton
Havana
High
Gary Kotton
Icehouse
High
Gary Kotton
OpenStack Security Advisory
Medium
Jeremy Stanley
VMwareAPI-Team
High
Unassigned

Bug Description

nova rescue of VM on vmWare will create a additional VM ($ORIGINAL_ID-rescue), but after that, the original VM has status ACTIVE. This leads to

[root@jhenner-node ~(keystone_admin)]# nova unrescue foo
ERROR: Cannot 'unrescue' while instance is in vm_state stopped (HTTP 409) (Request-ID: req-792cabb2-2102-47c5-9b15-96c74a9a4819)

the original can be deleted, which then causes leaking of the -rescue VM.

CVE References

Maithem (maithem) on 2014-01-16
summary: - nova rescue doesn't put VM into RESCUE status when using VMwareVCDriver
+ VMware: nova rescue doesn't put VM into RESCUE status when using
+ VMwareVCDriver

This doesn't seem to be a VC driver issue, I have tried this with the libvirt driver and the same problem happens. The state of the VM is not being set correctly.

Changed in nova:
status: New → Confirmed
summary: - VMware: nova rescue doesn't put VM into RESCUE status when using
- VMwareVCDriver
+ nova rescue doesn't put VM into RESCUE status
Maithem (maithem) on 2014-01-21
Changed in nova:
assignee: nobody → Maithem (maithem)
Thierry Carrez (ttx) wrote :

Looking into potential security implications.

information type: Public → Public Security
Changed in ossa:
status: New → Incomplete
Thierry Carrez (ttx) wrote :

I'm not convinced the leaking of the rescue VM is a security issue. Does it let the user circumvent its quota and the extra VMs could be used to overload compute nodes without generating any cost ?

Tracy Jones (tjones-i) on 2014-03-03
tags: added: compute
removed: vmware
John Garbutt (johngarbutt) wrote :

This seems specific to vmware, adding vmware tag.

tags: added: vmware
summary: - nova rescue doesn't put VM into RESCUE status
+ nova rescue doesn't put VM into RESCUE status on vmware

@John: comment #1 says libvirt is also affected ?

Tracy Jones (tjones-i) wrote :

@John - that's why i removed it because of comment #1

Tracy Jones (tjones-i) on 2014-03-04
Changed in openstack-vmwareapi-team:
status: New → Confirmed
Changed in nova:
importance: Undecided → High
Changed in openstack-vmwareapi-team:
importance: Undecided → High
Changed in nova:
milestone: none → icehouse-rc1
Jaroslav Henner (jhenner) wrote :

I have checked that this bug allows the user to pass the limits, so this can lead to DOS and also may lead to consuming more resources than has been paid for.

for i in {1..10}; do
  nova boot --image cirros-0.3.1-x86_64-disk.vmdk --flavor m1.micro foo-$i;
done

# Wait for them to spin-up to ACTIVE state. The limit is 10VMs. No more VMs can be spun up by thi tenant.

for i in {1..10}; do
  nova rescue foo-$i;
done

for i in {1..10}; do
  nova delete foo-$i;
done

# *-rescue machines are still there, in running state
# -- consuming RAM and disk space
# Goto the begining. We can spin-up 10 more VMs.

I have also checked that this problem is not present when using the libvirt/KVM VMs on local disk. The -rescue disk-file is created in the directory of the VM, which gets removed when the VM is being deleted. That holds for Grizzly and Havana release.

Grant Murphy (gmurphy) wrote :

If Jaroslav's analysis is correct than I think we should move to get a CVE assigned for this. (ttx do you agree?).

I've drafted an impact analysis (below). I'm not sure about the versions. I'm assuming this was introduced by this commit: 8db2f23a43cf8aff21d7ef07af742383c275dd76, however I will need a developer to confirm this problem and the cause.

------

Title: Nova VMWare driver leaks rescued images
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Versions: 2013.2

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova. By
requesting Nova place an image into rescue, then deleting
the image an authenticated user my exceed their quota. This can
result in a denial of service via excessive resource consumption.
Only setups using the Nova VMWare driver are affected.

Maithem (maithem) on 2014-03-13
Changed in nova:
assignee: Maithem (maithem) → nobody
Gary Kotton (garyk) wrote :

Correct, the problem is specific to VMware. More specifically if a instance ins deleted after a resuce then a rescus VM will still exist on the backend.

Changed in nova:
assignee: nobody → Gary Kotton (garyk)

Fix proposed to branch: master
Review: https://review.openstack.org/80284

Changed in nova:
status: Confirmed → In Progress
Thierry Carrez (ttx) on 2014-03-17
Changed in ossa:
status: Incomplete → Confirmed
assignee: nobody → Grant Murphy (gmurphy)
importance: Undecided → Medium

Impact description looks good. I would add a comma between "the image" and "an authenticated user", for clarity.
Also would be good to get confirmation on the affected versions.

Jeremy Stanley (fungi) wrote :

Grant, looks like it should be "Versions: from 2013.2 to 2013.2.2" (assuming it does not affect releases prior to Havana and will be fixed in the 2013.2.3 stable branch point release).

Thierry Carrez (ttx) on 2014-03-19
Changed in ossa:
status: Confirmed → Triaged
Tracy Jones (tjones-i) on 2014-03-19
Changed in openstack-vmwareapi-team:
status: Confirmed → In Progress
Grant Murphy (gmurphy) wrote :

@garyk Should this change also be backported to Havana?

Grant Murphy (gmurphy) on 2014-03-21
summary: - nova rescue doesn't put VM into RESCUE status on vmware
+ nova rescue doesn't put VM into RESCUE status on vmware (CVE-2014-2573)

We'll need an havana backport, but maybe wait for the patch to make it to master first to avoid duplication of work

Changed in ossa:
status: Triaged → In Progress
Thierry Carrez (ttx) on 2014-03-25
Changed in nova:
milestone: icehouse-rc1 → none
tags: added: icehouse-rc-potential
Grant Murphy (gmurphy) wrote :

This seems to be having some difficulties getting through the gate. Can we please get some more eyes on this?

Thierry Carrez (ttx) wrote :

master patch blocked on vmware CI. Will need to be backported to milestone-proposed and stable/havana

Thierry Carrez (ttx) on 2014-04-17
tags: added: icehouse-backport-potential
removed: icehouse-rc-potential

Reviewed: https://review.openstack.org/80284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=efb66531bc37ee416778a70d46c657608ca767af
Submitter: Jenkins
Branch: master

commit efb66531bc37ee416778a70d46c657608ca767af
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf
    Closes-bug: 1269418

Changed in nova:
status: In Progress → Fix Committed
Nanuk Krinner (nkrinner) wrote :

I'm working on a backport of the original patch written by Gary Kotton , currently running tests.

Thierry Carrez (ttx) wrote :

Above links are for stable/havana.
For referenc, here are the icehouse ones:
https://review.openstack.org/#/c/88514/ (now outdated)
https://review.openstack.org/#/c/89217/

Jeremy Stanley (fungi) on 2014-05-26
Changed in ossa:
assignee: Grant Murphy (gmurphy) → Jeremy Stanley (fungi)

Reviewed: https://review.openstack.org/89217
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ffcb17678c7e5409a1f12a09945b18e8879a677d
Submitter: Jenkins
Branch: stable/icehouse

commit ffcb17678c7e5409a1f12a09945b18e8879a677d
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Closes-bug: 1269418
    (cherry picked from commit efb66531bc37ee416778a70d46c657608ca767af)

    Conflicts:

     nova/virt/vmwareapi/vmops.py

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf

Reviewed: https://review.openstack.org/89768
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3cc3f62a60662e5bb82136c0cfa464592a6afe9
Submitter: Jenkins
Branch: stable/havana

commit b3cc3f62a60662e5bb82136c0cfa464592a6afe9
Author: Gary Kotton <email address hidden>
Date: Thu Mar 13 06:53:58 2014 -0700

    VMware: ensure rescue instance is deleted when instance is deleted

    If the user creates a rescue instance and then proceeded to delete
    the original instance then the rescue instance would still be up
    and running on the backend.

    This patch ensures that the rescue instance is cleaned up if
    necessary.

    The vmops unrescue method has a new parameter indicating if
    the original VM should be powered on.

    Closes-bug: 1269418
    (cherry picked from commit efb66531bc37ee416778a70d46c657608ca767af)

    Conflicts:

     nova/tests/virt/vmwareapi/test_vmwareapi.py
     nova/virt/vmwareapi/vmops.py

    Change-Id: I3c1d0b1d003392b306094b80ea1ac99377441fbf

Alan Pevec (apevec) on 2014-05-29
tags: removed: icehouse-backport-potential
Jeremy Stanley (fungi) on 2014-05-29
Changed in ossa:
status: In Progress → Fix Committed
Jeremy Stanley (fungi) on 2014-05-29
summary: - nova rescue doesn't put VM into RESCUE status on vmware (CVE-2014-2573)
+ [OSSA 2014-017] nova rescue doesn't put VM into RESCUE status on vmware
+ (CVE-2014-2573)
Changed in ossa:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-06-11
Changed in nova:
milestone: none → juno-1
status: Fix Committed → Fix Released
Jaroslav Henner (jhenner) wrote :
Download full text (3.4 KiB)

It fails to unrescue when the vmware VM is in powered-on state and also it seems to be leaking when the vm is in ERROR state.

# nova boot --image cirros-0.3.1-x86_64-disk.vmdk --flavor m1.tiny foo
# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ACTIVE | - | Running | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
# nova rescue foo
[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ACTIVE | rescuing | Running | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+

******* Wait until the -rescue VM gets powered on, then the nova seems to switch the state:

[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | RESCUE | - | Shutdown | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
# nova unrescue foo
# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | RESCUE | unrescuing | Shutdown | novanetwork=192.168.32.5 |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
[root@jhenner-node-permanent-7v0 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------+
| c1ed3110-a054-420c-bc70-0bdb254fcc8c | foo | ERROR | - | Shutdown | novanetwork=192.168....

Read more...

Grant Murphy (gmurphy) wrote :

This issue has been closed. Any incomplete fixes / regressions should be raised in a new (private) bug referencing this one.

Thierry Carrez (ttx) on 2014-10-16
Changed in nova:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.