[OSSA 2013-026] Some sequence of characters in console-log can DoS nova-compute (CVE-2013-4261)

Bug #1215091 reported by Thierry Carrez
256
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
High
Michael Still
Grizzly
Fix Released
High
Unassigned
OpenStack Security Advisory
Fix Released
High
Thierry Carrez
oslo-incubator
Invalid
Undecided
Unassigned
Folsom
Fix Committed
Undecided
Unassigned
Grizzly
Fix Committed
Undecided
Unassigned

Bug Description

Reported publicly by Jaroslav Henner at: https://bugzilla.redhat.com/show_bug.cgi?id=999164

for some sequence of characters in the console-log, nova console-log displays:
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

When console-log is ran often enough, it seems to be causeing death of nova-compute.

CVE References

Revision history for this message
Thierry Carrez (ttx) wrote :

mikal is working on fixing our old friend bug 832507, maybe he should take a look at this one as well.

Changed in ossa:
status: New → Incomplete
Revision history for this message
Michael Still (mikal) wrote :

This specific failure looks qpid specific, but I think we should probably be filtering to only printable characters for all transports. I will take a look at this ASAP.

Changed in nova:
importance: Undecided → Critical
assignee: nobody → Michael Still (mikalstill)
status: New → Triaged
importance: Critical → High
Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote :

I was debugging this issue and found out it is an instance of bug 1175808 only in grizzly. See https://bugzilla.redhat.com/show_bug.cgi?id=999164#c6

I'll propose the full backport of the patch in bug 1175808 to grizzly which fixes the issue.

Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote :

Proposed fix for grizzly: https://review.openstack.org/#/c/43303/

Revision history for this message
Thierry Carrez (ttx) wrote :

@Xavier: so havana (master) is unaffected ? What about Folsom (not sure QPID support existed back then...) ?

@Mikal: are you taking the OSSA task too ? (i.e. draft the impact description and communicate it around once all fixes are merged ?)

Changed in ossa:
status: Incomplete → Confirmed
importance: Undecided → High
Changed in nova:
status: Triaged → Incomplete
Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote :

Yes, the problem only appears in grizzly (and probably folsom) where the qpid implementation wasn't updated to send messages bigger than 65KB

An easy way to reproduce it is as follows:

1. Start a new instance
2. From inside the instance fill the console log (/dev/ttyS0) with as much text as possible (to make the log bigger than 65KB)
3. Ask for the full console-log (either from novaclient or horizon), which should fail with a 500 - InternalError
  - repeat #3 for at least "rpc_conn_pool_size" times until the call hangs.
4. nova-compute has no more connections in the pool and cannot be accessed

Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote :

NOTE: rpc_conn_pool_size default is 30

Revision history for this message
Michael Still (mikal) wrote :

I can confirm that I cannot duplicate this problem on trunk / rabbitmq.

Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → Invalid
Changed in oslo:
status: New → Invalid
Revision history for this message
Thierry Carrez (ttx) wrote :

Proposed impact description:

"""
Title: Potential denial of service on Nova when using Qpid
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Affects: Folsom, Grizzly

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova when using Apache Qpid as the RPC backend. By sending malicious characters to an instance console and requesting the console log contents through the API, an authenticated user may disrupt the nova-compute node his instance is running on. This vulnerability could be leveraged in a Denial of Service attack against the cloud provider. Only Folsom and Grizzly setups using Qpid as their RPC backend are affected. Havana setups, or setups using other RPC backends (like RabbitMQ), are all unaffected.
"""

Changed in ossa:
assignee: nobody → Thierry Carrez (ttx)
status: Confirmed → Triaged
Revision history for this message
Thierry Carrez (ttx) wrote :

Trick question: are Cinder and Neutron (which also have the option to use Qpid as their rpc backend) vulnerable to the same DoS issue ? Or are they protected because they don't have an equivalent of nova console-log ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/44700

Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote : Re: Some sequence of characters in console-log can DoS nova-compute

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/44695

Revision history for this message
Thierry Carrez (ttx) wrote :

@Xavier: could you have a look at the impact description and check it makes sense ? Also if you have the answer to the trick question in comment 10...

Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote :

@Thierry: I would only replace the part saying "malicious characters" with something like "any random text longer than 65K characters".

w.r.t. comment 10, I guess there shouldn't be a problem in other projects while we don't send user generated content like the console log but I should double check this as I am not familiar with cinder and neutron internals.

Revision history for this message
Thierry Carrez (ttx) wrote :

New impact description:

"""
Title: Potential denial of service on Nova when using Qpid
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Affects: Folsom, Grizzly

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova when using Apache Qpid as the RPC backend. By sending any random text longer than 65K characters to an instance console and requesting the console log contents through the API, an authenticated user may disrupt the nova-compute node his instance is running on. This vulnerability could be leveraged in a Denial of Service attack against the cloud provider. Only Folsom and Grizzly setups using Qpid as their RPC backend are affected. Havana setups, or setups using other RPC backends (like RabbitMQ), are all unaffected.
"""

I'll try to get Cinder/neutron folks to look into the issue and confirm if they are affected.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (stable/grizzly)

Reviewed: https://review.openstack.org/44695
Committed: http://github.com/openstack/oslo-incubator/commit/4f97479ad078771f6f25461c95203a5d293ec08b
Submitter: Jenkins
Branch: stable/grizzly

commit 4f97479ad078771f6f25461c95203a5d293ec08b
Author: Xavier Queralt <email address hidden>
Date: Mon Sep 2 14:27:22 2013 +0200

    Fix problem when sending long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    The first part of this fix was ported to Grizzly in I8b6c5734b to allow
    receiving messages from Havana using the new format. Even though this
    change will modify the message format, it will only do it when messages
    are longer than 65K which would be broken anyway and could cause serious
    bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (stable/folsom)

Reviewed: https://review.openstack.org/44700
Committed: http://github.com/openstack/oslo-incubator/commit/478ac3a3ec4b2dd9adb32891123b6e33c483bdf2
Submitter: Jenkins
Branch: stable/folsom

commit 478ac3a3ec4b2dd9adb32891123b6e33c483bdf2
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

    Fix problem with long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    Even though this change will modify the message format, it will only do
    it when messages are longer than 65K which would be broken anyway and
    could cause serious bugs like the one linked below.

    Fixes bug 1215091

    (cherry picked from commit 7ce54410485b33cffc39c7ffb96eae422b88601c)

    Conflicts:
     openstack/common/rpc/impl_qpid.py

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/45426

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Some sequence of characters in console-log can DoS nova-compute

Xavier: could you clarify which one of those patches is actually the CVE fix ? I'm a bit confused (and admittedly tired too :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/45426
Committed: http://github.com/openstack/nova/commit/ef5730a4620b409a3b46e46966e3bc6f3a306464
Submitter: Jenkins
Branch: stable/folsom

commit ef5730a4620b409a3b46e46966e3bc6f3a306464
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

    Fix problem with long messages in Qpid (from oslo)

    This is commit 478ac3a3e in oslo-incubator

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    Even though this change will modify the message format, it will only do
    it when messages are longer than 65K which would be broken anyway and
    could cause serious bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

Revision history for this message
Xavier Queralt (xqueralt-deactivatedaccount) wrote : Re: Some sequence of characters in console-log can DoS nova-compute

@Thierry: the patches that fix the CVE are the ones in nova:

https://review.openstack.org/#/c/45426/ (for folsom) and https://review.openstack.org/#/c/43303/ (for grizzly)

The other two are just the patches to oslo-incubator.

Thierry Carrez (ttx)
Changed in ossa:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/43303
Committed: http://github.com/openstack/nova/commit/2d949c415b97ed9649e78c880ab149d0d39c1152
Submitter: Jenkins
Branch: stable/grizzly

commit 2d949c415b97ed9649e78c880ab149d0d39c1152
Author: Xavier Queralt <email address hidden>
Date: Thu Sep 5 10:08:29 2013 +0200

    Fix Qpid when sending long messages (from oslo)

    This is commit 4f97479ad in oslo-incubator

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    The first part of this fix was ported to Grizzly in Ib52e9458a to allow
    receiving messages from Havana using the new format. Even though this
    change will modify the message format, it will only do it when messages
    are longer than 65K which would be broken anyway and could cause serious
    bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I505b648c3d0e1176ec7a3fc7d1646fa5a5232261

Thierry Carrez (ttx)
Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote :

[OSSA 2013-026]

Changed in ossa:
status: Fix Committed → Fix Released
summary: - Some sequence of characters in console-log can DoS nova-compute
+ [OSSA 2013-026]Some sequence of characters in console-log can DoS nova-
+ compute (CVE-2013-4261)
summary: - [OSSA 2013-026]Some sequence of characters in console-log can DoS nova-
+ [OSSA 2013-026] Some sequence of characters in console-log can DoS nova-
compute (CVE-2013-4261)
Sean Dague (sdague)
no longer affects: nova/folsom
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.