[OSSA 2013-037] DoS attack via setting os_type in snapshots (CVE-2013-6437)

Bug #1253980 reported by Phil Day
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
moorryan
Grizzly
Fix Released
High
moorryan
Havana
Fix Released
High
moorryan
OpenStack Security Advisory
Fix Released
High
Thierry Carrez

Bug Description

If the os_type metadata is set of an image, the ephemeral disk backing file for that image will be named ephemeral_[size]_[os_type].
Because the user can change os_type they can use this to create new ephemeral backing files.
Nova image cache management does not include deleting ephemeral backing files (presumably because they are expected to be a small, stable set.

Hence a user can fill the disk with ephemeral backing files via the following:

1) Spawn a instance
2) Create a snapshot from it, delete the original instance
3) In a loop:
generate a random os_type
set os_type to the snapshot
spawn and instance from it, and then delete the instance

Every iteration will generate an ephemeral backing file on a compute host. With a stacking scheduling policy there is a good chance of hitting the same host repeatedly until its disk is full.

Suggested mitigation

Only use “os_type” in the ephemeral file name if there is a specific mkfs command defined, otherwise use “default” (Currently for undefined os-types it will use the default mkfs command, but still uses os_type in the name.

CVE References

Phil Day (philip-day)
description: updated
Revision history for this message
Thierry Carrez (ttx) wrote :

Sounds legit

Changed in ossa:
status: New → Incomplete
Revision history for this message
Russell Bryant (russellb) wrote :

Yep, seems legit to me, too.

Revision history for this message
Andrew Laski (alaski) wrote :

I went looking through the code to figure out any implications of changing the backing file name and honestly I'm lost in image caching code, but as far as I can tell the suggested fix should work and would be fairly easy.

Phil, are you going to submit a patch for this?

Revision history for this message
Phil Day (philip-day) wrote :

Don't have anyone working on it yet, but was hoping I could get someone started this week. Open to other offers though

Revision history for this message
Andrew Laski (alaski) wrote :

I'm out for the rest of the week and have a lot on my plate already. But if this sits too long I may try to tackle it.

Jeremy Stanley (fungi)
Changed in ossa:
status: Incomplete → Confirmed
Revision history for this message
Phil Day (philip-day) wrote :

Added Ryan Moor (HP) as he is going to start working on a patch for this

Thierry Carrez (ttx)
Changed in ossa:
importance: Undecided → High
Revision history for this message
moorryan (moorryan) wrote :

Attached is a patch that contains a solution

Never submitted tin this way before so not sure what is required

Commit message: >>
use 'os_type' in ephemeral filename only if mkfs defined

Currently for undefined os-types it will use the default mkfs command, but use the meta 'os_type' in the name of the ephemeral file (e.g. ephemeral_20_abcdef). Which can result in a lot of files (DoS?)
This change will only use 'os_type' in the ephemeral filename if there is a specific mkfs command defined, otherwise it will use 'default' (e.g. ephemeral_20_default)

Modifed the tests to test for:
  os_type=''
  os_type=None
  os_type='test' - with no mkfs command specified
  os_type='test' - with a mkfs command specified
<<

run it up in devstack and it behaves as expected

moorryan (moorryan)
Changed in ossa:
assignee: nobody → moorryan (moorryan)
status: Confirmed → In Progress
Phil Day (philip-day)
Changed in nova:
status: New → In Progress
importance: Undecided → High
assignee: nobody → moorryan (moorryan)
Revision history for this message
Andrew Laski (alaski) wrote :

The patch from Ryan looks good to me and should address the reported issue.

Revision history for this message
Phil Day (philip-day) wrote :

Patch looks good to me too - nice work.

The patch doesn't seem to apply on stable/havana though - so I guess we need a separate patch file for that.

Any others we need ?

Revision history for this message
Thierry Carrez (ttx) wrote :

We'll need a stable/havana patch for this. I suspect this affects grizzly too so we'll need a stable/grizzly patch too.

Changed in ossa:
assignee: moorryan (moorryan) → nobody
status: In Progress → Confirmed
Revision history for this message
Russell Bryant (russellb) wrote :

master patch looks good to me, too. Thanks!

So we just need the stable backports now. moorryan, do you think you can work on some backports, too?

Revision history for this message
moorryan (moorryan) wrote :

backported for stable/havana

Revision history for this message
moorryan (moorryan) wrote :

backported for stable/grizzly

Revision history for this message
Phil Day (philip-day) wrote :

The stable patches look good to me - nice work Ryan.

Revision history for this message
Andrew Laski (alaski) wrote :

All 3 patches look good to me. Thanks Ryan.

Revision history for this message
Thierry Carrez (ttx) wrote :

Proposed impact description:

Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the handling of ephemeral disk backing files on Nova compute nodes. By repeatedly spawning new instances of a snapshotted server with random os_type (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup.

Changed in ossa:
assignee: nobody → Thierry Carrez (ttx)
status: Confirmed → Triaged
Revision history for this message
Thierry Carrez (ttx) wrote :

After writing the impact description it appears to me that this DoS vector could end up being costly for the attacker. I tend to consider "costly DoS" to fall in the same category as "normal usage" for a resource provider... Thoughts on that ? Should this still be considered a practical vulnerability ?

Revision history for this message
Jeremy Stanley (fungi) wrote :

In my opinion, it comes down to whether there is typically a quota/rate-limit around these sorts of operations, and whether or not they are likely to be tracked/billed by a provider. But I don't have the answer to those (hopefully someone here with OpenStack operational background can confirm).

Revision history for this message
Phil Day (philip-day) wrote : RE: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
Download full text (3.3 KiB)

Hi Thierry,

A minor clarification on the impact description:

"By repeatedly spawning new instances of a snapshotted server with random os_type ..." should read " By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot ..."

I don’t think its necessarily that costly - you do have to pay for each instance you create, but it can be the smallest instance needed and doesn’t have to exist any longer than the time to create it. You could probably even kill the instance before it gets to Active (It only needs to get past the creation of the ephemeral backing file) and avoid any billing at all on some systems. Likewise you don't have to keep the snapshots once the instance has been created. Because the ephemeral backing files are never cleared up (its assumed that they are small and a limited set) you don't have to do this at a fast rate to create a long term attack.

Phil

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 09 December 2013 16:06
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> After writing the impact description it appears to me that this DoS vector
> could end up being costly for the attacker. I tend to consider "costly DoS" to
> fall in the same category as "normal usage" for a resource provider...
> Thoughts on that ? Should this still be considered a practical vulnerability ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral backing file on a compute
> host. With a stacking scheduling policy there is a good chance of
> hitting the same host repeatedly until its disk is full.
>
> Suggested mitigation
>
> Only use “os_type” in the ephemeral file name if there is a specific
> mkfs command defined, otherwise use “default” (Currently for
> undefined os-types it will use the default mkfs command, but still
> uses os_...

Read more...

Revision history for this message
Thierry Carrez (ttx) wrote : Re: DoS attack via setting os_type in snapshots.

New proposed impact desc:

--------------------------------------------
Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the handling of ephemeral disk backing files on Nova compute nodes. By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup.
--------------------------------------------

@Phil: agreed... waiting for confirmation from other VMT members to go ahead. Would the ephemeral backing files all land on the same server (reaching a state of partial DoS quickly) or be spread across all compute nodes (reaching a state of total DoS but slowly) ?

Revision history for this message
Phil Day (philip-day) wrote : RE: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
Download full text (3.7 KiB)

> @Phil: agreed... waiting for confirmation from other VMT members to go
> ahead. Would the ephemeral backing files all land on the same server
> (reaching a state of partial DoS quickly) or be spread across all compute
> nodes (reaching a state of total DoS but slowly) ?

That depends entirely on the scheduling policy configured in Nova - a stacking model would hit the issue on specific nodes more quickly that a spreading model (so a quicker DoS on specific servers) - but really it’s the total DoS that is the issue which is kind of the same in either case, its just a question of if you fill from the side or from the bottom.

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 10 December 2013 09:49
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> New proposed impact desc:
>
> --------------------------------------------
> Title: Nova compute DoS through ephemeral disk backing files
> Reporter: Phil Day (HP)
> Products: Nova
> Affects: All supported versions
>
> Description:
> Phil Day from HP reported a vulnerability in the handling of ephemeral disk
> backing files on Nova compute nodes. By repeatedly creating snapshots,
> changing the os_type to a new random value, and spawning new instances
> from the snapshot (and quickly deleting those instances), an authenticated
> user could generate lots of different ephemeral disk backing files and fill up
> compute node disks, potentially resulting in a Denial of Service against a Nova
> setup.
> --------------------------------------------
>
> @Phil: agreed... waiting for confirmation from other VMT members to go
> ahead. Would the ephemeral backing files all land on the same server
> (reaching a state of partial DoS quickly) or be spread across all compute
> nodes (reaching a state of total DoS but slowly) ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral back...

Read more...

Revision history for this message
Andrew Laski (alaski) wrote : Re: DoS attack via setting os_type in snapshots.

Should the proposed impact mention that this is libvirt specific? Otherwise it lgtm.

Revision history for this message
Thierry Carrez (ttx) wrote :

If it is libvirt-specific, then yes it should mention it. Is this the case ?

Revision history for this message
Andrew Laski (alaski) wrote :

This issue is libvirt specific. Other drivers may or may not have similar issues but the code paths are very different and can't be generalized under this bug. So adding " running the libvirt driver." or something similar to the first sentence should cover it.

Revision history for this message
Thierry Carrez (ttx) wrote :

OK, how about:

--------------------------------------------
Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the libvirt driver handling of ephemeral disk backing files on Nova compute nodes. By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup. Only Nova setups running the libvirt driver are affected.
--------------------------------------------

Revision history for this message
Phil Day (philip-day) wrote : RE: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.

LGTM - thanks Thierry

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 11 December 2013 11:20
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> OK, how about:
>
> --------------------------------------------
> Title: Nova compute DoS through ephemeral disk backing files
> Reporter: Phil Day (HP)
> Products: Nova
> Affects: All supported versions
>
> Description:
> Phil Day from HP reported a vulnerability in the libvirt driver handling of
> ephemeral disk backing files on Nova compute nodes. By repeatedly creating
> snapshots, changing the os_type to a new random value, and spawning new
> instances from the snapshot (and quickly deleting those instances), an
> authenticated user could generate lots of different ephemeral disk backing
> files and fill up compute node disks, potentially resulting in a Denial of Service
> against a Nova setup. Only Nova setups running the libvirt driver are
> affected.
> --------------------------------------------
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral backing file on a compute
> host. With a stacking scheduling policy there is a good chance of
> hitting the same host repeatedly until its disk is full.
>
> Suggested mitigation
>
> Only use “os_type” in the ephemeral file name if there is a specific
> mkfs command defined, otherwise use “default” (Currently for
> undefined os-types it will use the default mkfs command, but still
> uses os_type in the name.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1253980/+subscriptions

Revision history for this message
Andrew Laski (alaski) wrote : Re: DoS attack via setting os_type in snapshots.

Lgtm as well.

Revision history for this message
Thierry Carrez (ttx) wrote :

CVE requested

Changed in ossa:
status: Triaged → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote :

CVE-2013-6437

summary: - DoS attack via setting os_type in snapshots.
+ DoS attack via setting os_type in snapshots (CVE-2013-6437)
Revision history for this message
Thierry Carrez (ttx) wrote : Re: DoS attack via setting os_type in snapshots (CVE-2013-6437)

Proposed public disclosure date/time: Wednesday, December 18, 1500UTC.

Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote :

Tested:

stable/grizzly pep8: PASS
stable/grizzly py27: PASS
stable/grizzly dsvm-full: PASS
stable/grizzly dsvm-large-ops: PASS
stable/havana pep8: PASS
stable/havana py27: PASS
stable/havana dsvm-full: PASS
stable/havana dsvm-neutron: PASS
master pep8: PASS
master py27: PASS

Revision history for this message
Thierry Carrez (ttx) wrote :
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/62910
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6e455cd97f04bf26bbe022be17c57e089cf502f4
Submitter: Jenkins
Branch: master

commit 6e455cd97f04bf26bbe022be17c57e089cf502f4
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/62912
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ca38774ebcf5b67d16c202c8f218c0c433973ca9
Submitter: Jenkins
Branch: stable/havana

commit ca38774ebcf5b67d16c202c8f218c0c433973ca9
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Backport of Original Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c
    to stable/havana

    Change-Id: Ifa2b94e79dabd586d7e904da247d099360229313

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/62913
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e451f1bac57d24e47171cffb3ad59bb1610d836
Submitter: Jenkins
Branch: stable/grizzly

commit 3e451f1bac57d24e47171cffb3ad59bb1610d836
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Backport of Original Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c
    to stable/grizzly

    Change-Id: Ia040910e90861a2987eff345ad1c01863655b124

Revision history for this message
Thierry Carrez (ttx) wrote :

OSSA 2013-037

Changed in ossa:
status: Fix Committed → Fix Released
summary: - DoS attack via setting os_type in snapshots (CVE-2013-6437)
+ [OSSA 2013-037] DoS attack via setting os_type in snapshots
+ (CVE-2013-6437)
Changed in nova:
milestone: none → icehouse-2
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-2 → 2014.1
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.