Specific config setting may result in VMs being taken over through VNC

[Thierry Carrez]

Jonathan Hogg from Chargebox reports (edited):

On a single-machine cloud running OpenStack Icehouse and over the last week we have seen compromises of all of the Ubuntu 14.04 VMs running on the machine. Scenario shows the attacker gaining access through VNC (via controlled reboot to reset root password).

QEMU instances are running with -vnc 0.0.0.0:1, which may or may not be the issue.

[Thierry Carrez (ttx)]

The single-machine cloud was not firewalled, so if QEMU instance exposes the VNC console on 0.0.0.0:1, attackers could find that one through portscan and exploit it.

[Thierry Carrez (ttx)]

Still sounds like a good idea to determine if that 0.0.0.0 is the default and if yes, if that could not be strengthened.

[Andrew Laski (alaski)]

There's not enough information here for me to fully understand what happened and what might need to be addressed.

Did the attacker gain access to VNC by using the nova-novncproxy, nova-xvpvncproxy, or other proxy? How was the authentication mechanism bypassed in order to gain a VNC session with an instance? What was rebooted, and via what means?

[Jeremy Stanley (fungi)]

From the information they provided, qemu was configured to start a VNC service listening on all interfaces, and the compute node's IP address was exposed with that socket unfiltered. It sounds like attackers scanning for VNC servers on the Internet found it, rebooted the virtual machines via ctrl-alt-del and then rooted them by altering bootloader configuration to boot into a shell rather than init. Their evidence suggests the connections were directly to qemu, not via the Nova VNC proxy at all.

[Andrew Laski (alaski)]

Makes sense, thanks. It wasn't totally clear to me if the compute node was exposing VNC via a public interface. It seems to me that this is mostly about recommending and documenting best practices for VNC setups. While looking for documentation on the appropriate config option I found https://lists.launchpad.net/openstack/msg22376.html which mentions this same type of issue. I can't find a direct link to the config option "vncserver_listen" but it defaults to 127.0.0.1. However, this guide http://docs.openstack.org/havana/install-guide/install/yum/content/nova-compute.html recommends setting it to 0.0.0.0.

[Sean Dague (sdague)]

Could you please post relevant nova.conf? If this is a user changed variable that makes this exploitable, I'm not sure it's a real sec bug.

[Sean Dague (sdague)]

Without more information, I agree with Jeremy, this is an Invalid bug. Users can configure themselves many ways which open up sec holes.

[Jeremy Stanley (fungi)]

Agreed, we should mark this invalid and switch it to public.

[Thierry Carrez (ttx)]

That said, documentation recommends setting up 0.0.0.0 so that "live migration can work":

http://docs.openstack.org/admin-guide-cloud/content/section_configuring-compute-migrations.html
"You must specify vncserver_listen=0.0.0.0 or live migration will not work correctly."

http://docs.openstack.org/admin-guide-cloud/content/nova-vncproxy-replaced-with-nova-novncproxy.html
"To connect the service to your Compute deployment, add the following configuration options to your nova.conf file:
    vncserver_listen=0.0.0.0"
"To use live migration, use the 0.0.0.0 address."

[Thierry Carrez (ttx)]

Opening up and moving to docs bug

[Anne Gentle (annegentle)]

Change these docs to tell users not to set to 0.0.0.0 and instead set to the management IP address of the controller node.

I'm not sure if that setting would still enable live migration, however, can anyone on this thread offer advice?

[Alexandra Settle (alexandra-settle)]

Okay, we definitely still document this: "You must specify vncserver_listen=0.0.0.0 or live migration will not work correctly." Which you can now find here: http://docs.openstack.org/admin-guide/compute-configuring-migrations.html

Marking as incomplete until someone on this thread can offer advice on how to edit this setting appropriately by answering the above question.

Thank you,

Alex

[Anne Gentle (annegentle)]

Here's what I learned, asking engineering and operations folks. Yes, the setting must be 0.0.0.0 to enable migration. That said, best practice is to ensure that iptables rules protect the VNC proxy server.

Assigning to myself to add an additional best practice bullet point.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/433321
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=d192e0d2a83ec8dd58ca96d3f3f112ce02fbff8a
Submitter: Jenkins
Branch: master

commit d192e0d2a83ec8dd58ca96d3f3f112ce02fbff8a
Author: Anne Gentle <email address hidden>
Date: Mon Feb 13 16:53:03 2017 -0600

    Adds additional best practice security measures for live migration VNC proxy settings

    Fixes-bug: 1435386
    Change-Id: Ib7a6e7691d9f1106d0073115d1582415181b3fbc