Docs recommend insecure configuration option

Bug #1287194 reported by Thierry Carrez on 2014-03-03
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Undecided
Unassigned
OpenStack Security Notes
High
Nathan Kinder
openstack-manuals
Critical
Anne Gentle

Bug Description

From Daniel Berrange:

---------
I just came across the following docs for configuring migration with
nova+libvirt+kvm

  http://docs.openstack.org/trunk/config-reference/content//section_configuring-compute-migrations.html
  http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-migrations.html

At point 7 the docs say

 7. Update the libvirt configurations. Modify the /etc/libvirt/libvirtd.conf file:

  before : #listen_tls = 0
  after : listen_tls = 0
  before : #listen_tcp = 1
  after : listen_tcp = 1
  add: auth_tcp = "none"

  Modify the /etc/init/libvirt-bin.conf file:

  before : exec /usr/sbin/libvirtd -d
  after : exec /usr/sbin/libvirtd -d -l

What this does is tell the libvirt daemon listen for client TCP connections
on all network interfaces, and accept any clients performing absolutely
zero authentication.

Sure this works for migration, but it also allows anyone of the network
to be able to completely own all your compute hosts, by invoking whatever
libvirt API calls they like.

This is equivalent to telling someone to configure SSH to allow root
logins with no passwords or keys at all. Actually it is worse because
as well as killing authentication, it kills any encryption too.

Libvirt has a choice of 4 secure options for remote access over TCP

 - SSH tunnel to libvirtd's UNIX socket
 - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
 - libvirtd TCP socket, with TLS for encryption and x509 client
   certs for authentication
 - libvirtd TCP socket, with TLS for encryption and Kerberos for
   authentication

documenting any of these setups would be better than what's there
now, which needs to be removed asap.

I don't know whether these docs are the current supported / preferred
docs for this - they're just what I found via google. If we have other
docs covering migration setup, they should be checked too.
----------------

Thierry Carrez (ttx) on 2014-03-03
Changed in ossa:
status: New → Incomplete
Revision history for this message
Thierry Carrez (ttx) wrote :

This is not something that gets fixed by upgrading, so not OSSA territory. I would very much like to see an OSSN about this though.

I would also argue that making this public ASAP would help more than it hurts.

Changed in ossa:
status: Incomplete → Won't Fix
Revision history for this message
Tom Fifield (fifieldt) wrote :

does anyone subscribed to the bug have time to make a test system to verify instructions for one of the other configuration methods?

Revision history for this message
Anne Gentle (annegentle) wrote : Re: [Bug 1287194] Re: Docs recommend insecure configuration option

Do you think that step-by-step instructions are required? I don't since
there are considerations based on environment that may cause people to
choose. I've submitted three patches that are not step-by-step.

On Mon, Mar 3, 2014 at 9:14 AM, Tom Fifield <email address hidden>wrote:

> does anyone subscribed to the bug have time to make a test system to
> verify instructions for one of the other configuration methods?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1287194
>
> Title:
> Docs recommend insecure configuration option
>
> Status in OpenStack Manuals:
> New
> Status in OpenStack Security Advisories:
> Won't Fix
> Status in OpenStack Security Notes:
> New
>
> Bug description:
> From Daniel Berrange:
>
> ---------
> I just came across the following docs for configuring migration with
> nova+libvirt+kvm
>
>
> http://docs.openstack.org/trunk/config-reference/content//section_configuring-compute-migrations.html
>
> http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-migrations.html
>
> At point 7 the docs say
>
> 7. Update the libvirt configurations. Modify the
> /etc/libvirt/libvirtd.conf file:
>
> before : #listen_tls = 0
> after : listen_tls = 0
> before : #listen_tcp = 1
> after : listen_tcp = 1
> add: auth_tcp = "none"
>
> Modify the /etc/init/libvirt-bin.conf file:
>
> before : exec /usr/sbin/libvirtd -d
> after : exec /usr/sbin/libvirtd -d -l
>
> What this does is tell the libvirt daemon listen for client TCP
> connections
> on all network interfaces, and accept any clients performing absolutely
> zero authentication.
>
> Sure this works for migration, but it also allows anyone of the network
> to be able to completely own all your compute hosts, by invoking whatever
> libvirt API calls they like.
>
> This is equivalent to telling someone to configure SSH to allow root
> logins with no passwords or keys at all. Actually it is worse because
> as well as killing authentication, it kills any encryption too.
>
> Libvirt has a choice of 4 secure options for remote access over TCP
>
> - SSH tunnel to libvirtd's UNIX socket
> - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
> - libvirtd TCP socket, with TLS for encryption and x509 client
> certs for authentication
> - libvirtd TCP socket, with TLS for encryption and Kerberos for
> authentication
>
> documenting any of these setups would be better than what's there
> now, which needs to be removed asap.
>
> I don't know whether these docs are the current supported / preferred
> docs for this - they're just what I found via google. If we have other
> docs covering migration setup, they should be checked too.
> ----------------
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1287194/+subscriptions
>

Revision history for this message
Nathan Kinder (nkinder) wrote :

@fifieldt
I did set up Havana (RDO distribution) with two compute nodes configured for live migration. I configured libvirtd to use the SASL GSSAPI mechanism to perform Kerberos authentication over an normal TCP connection (no TLS). Performing a live migration with the "nova live-migration" command works fine with this approach. It's not ideal, as you need to ensure that root has a Kerberos ticket first, which requires one to run kinit to get a valid ticket. This could likely be partially automated by setting the KRB5CCNAME environment variable in /etc/sysconfig/libvirtdand using a keytab to call kinit from a script, but one would still have to deal with ticket expiration.

I haven't yet tried the X.509 approach, but I don't expect any big issues there. In addition to the libvirtd configuration changes to use TLS for remote connections and X.509 client certificate authentication, the "live_migration_uri" setting in /etc/nova/nova.conf will need to be changed as follows:

    live_migration_uri=qemu+tls://%s/system

This change would also be needed if one were using GSSAPI/Kerberos over TLS.

One advantage with the X.509 approach is that certificates usually have a long lifetime. This makes expiration issues less of an issue since it's infrequent (usually every few years in most PKI deployments). One would simply be able to drop the client public and private key files onto each compute node (protected by the appropriate permissions), which would then be used for authentication between compute nodes during live migration. I will test this approach tomorrow, but I think this is enough to go on for putting together an OSSN.

Nathan Kinder (nkinder) on 2014-03-04
Changed in ossn:
assignee: nobody → Nathan Kinder (nkinder)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Nathan Kinder (nkinder) wrote :
Download full text (7.3 KiB)

Here is the draft for an OSSN for this issue. Please review it and provide any feedback.

------------------------------------------------------------------------------------------

Live migration instructions recommend unsecured libvirt remote access
---

### Summary ###
When using the KVM hypervisor with libvirt on OpenStack Compute nodes, live
migration of instances from one Compute server to another requires that the
libvirt daemon is configured for remote network connectivity. The libvirt
daemon configuration recommended in the OpenStack Configuration Reference
manual configures libvirtd to listen for incoming TCP connections on all
network interfaces without requiring any authentication or using any
encryption. This insecure configuration allows for anyone with network access
to the libvirt daemon TCP port on OpenStack Compute nodes to control the
hypervisor through the libvirt API.

### Affected Services / Software ###
Nova, Compute, KVM, libvirt, Grizzly, Havana, Icehouse

### Discussion ###
The default configuration of the libvirt daemon is to not allow remote access.
Live migration of running instances between OpenStack Compute nodes requires
libvirt daemon remote access between OpenStack Compute nodes.

The libvirt daemon should not be configured to allow unauthenticated remote
access. The libvirt daemon has a choice of 4 secure options for remote access
over TCP. These options are:

 - SSH tunnel to libvirtd's UNIX socket
 - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
 - libvirtd TCP socket, with TLS for encryption and x.509 client certificates
   for authentication
 - libvirtd TCP socket, with TLS for encryption and Kerberos for authentication

It is not necessary for the libvirt daemon to listen for remote TCP connections
on all interfaces. Remote network connectivity to the libvirt daemon should be
restricted as much as possible. Remote access is only needed between the
OpenStack Compute nodes, so the libvirt daemon only needs to listen for remote
TCP connections on the interface that is used for this communication. A
firewall can be configured to lock down access to the TCP port that the libvirt
daemon listens on, but this does not sufficiently protect access to the libvirt
API. Other processes on a remote OpenStack Compute node might have network
access, but should not be authorized to remotely control the hypervisor on
another OpenStack Compute node.

### Recommended Actions ###
If you are using the KVM hypervisor with libvirt on OpenStack Compute nodes,
you should review your libvirt daemon configuration to ensure that it is not
allowing unauthenticated remote access.

Remote access to the libvirt daemon via TCP is configured by the "listen_tls",
"listen_tcp", and "auth_tcp" configuration directives. By default, these
directives are all commented out. This results in remote access via TCP being
disabled.

If you do not need remote libvirt daemon access, you should ensure that the
following configuration directives are set as follows in the
/etc/libvirt/libvirtd.conf configuration file. Commenting out these directives
will have the same effect, as these values match the internal defaults...

Read more...

Revision history for this message
Nathan Kinder (nkinder) wrote :

I've made some slight changes to the last section of the "Recommended Actions" section of the OSSN draft. Specifically, I corrected an error in the ephemeral migration port range. I also added details on how to look up what ports are configured for libvirt remote access and live migration in case one is not using the defaults. Here is the new content:

-----------------------------------------------------------------------------
It is also recommended to configure the firewall on each OpenStack Compute node
to only allow other Compute nodes to access the ports that are used for remote
access to the libvirt daemon. By default, this is port 16514 for TLS, 16509
for unencrypted TCP, and an ephemeral port range of 49152-49215. You can check
what ports you have configured for the libvirt daemon by looking at the
following configuration directives:

  tls_port (libvirtd.conf)
  tcp_port (libvirtd.conf)
  migration_port_min (qemu.conf)
  migration_port_max (qemu.conf)

Please consult the documentation for your firewall software for instructions on
configuring the appropriate firewall rules.
-----------------------------------------------------------------------------

Revision history for this message
Daniel Berrange (berrange) wrote :

This looks like a pretty comprehensive and accurate writeup of the situation from libvirt POV

Revision history for this message
Nathan Kinder (nkinder) wrote :
Download full text (4.1 KiB)

I added some additional content to the "Recommended Actions" section. The additional content covers configuring the list address for the epehmeral migration ports, the ability to tunnel migration traffic over the remote access port, and details about securing the live block migration traffic.

I believe this gives a more complete picture of how live migration can/should be secured. It also will allow this single OSSN to cover the issues described in bug 1240554 instead of having two separte (but similar) OSSNs.

The new/reworked sections are below. Feedback would be appreciated.

----------------------------------------------------------------------
In addition to requiring authentication for remote access to the libvirt daemon
on your OpenStack Compute nodes, it is also recommended to restrict network
access such that connectivity is only allowed between the Compute nodes.

The first thing that should be done is to restrict the network interfaces that
the libvirt daemon listens on for remote connections. By default, the libvirt
daemon listens on all interfaces when remote access is enabled. This can be
restricted by setting the following configuration directive in
/etc/libvirt/libvirtd.conf:

---- begin example libvirtd.conf snippet ----
listen_addr = <IP address or hostname>
---- end example libvirtd.conf snippet ----

Migration in the libvirt daemon also uses a range of ephemeral ports by
default. The connections to these ephemeral ports are not authenticated or
encrypted. It is possible to tunnel the migration traffic over the regular
libvirt daemon remote access port, which will use the authentication and
encryption settings that you have defined for that port. It is recommended
that you do this for the additional security that it provides. To enable
tunneling of the migration traffic, you must tell your OpenStack Compute
nodes to set the VIR_MIGRATE_TUNNELLED flag for live migration. This is done
by setting the following directive in /etc/nova/nova.conf:

---- begin example nova.conf snippet ----
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_TUNNELLED
---- end example nova.conf snippet ----

The tunneling of migration traffic described above does not apply to live block
migration. Live block migration is currently only possible over ephemeral
ports.

If you choose to use the ephemeral migration ports, there are a few things that
you should be aware of. Unfortunately, there is no way to restrict the network
interfaces that these ephemeral ports will listen on in libvirt versions prior
to 1.1.4. If you are running version 1.1.4 or later of the libvirt daemon, you
can set the following directive in /etc/libvirt/qemu.conf to specify what
interfaces are used for the ephemeral migration ports:

---- begin example qemu.conf snippet ----
migration_address = <IP address>
---- end example qemu.conf snippet ----

It is also recommended to configure the firewall on each OpenStack Compute node
to only allow other Compute nodes to access the ports that are used for remote
access to the libvirt daemon. By default, this is port 16514 for TLS and 16509
for unencrypted TCP.

Additionally, migration over...

Read more...

Revision history for this message
Anne Gentle (annegentle) wrote :
Download full text (7.2 KiB)

I've read all the drafts and changes and I don't have any suggestions,
looks great!

Is the doc change sufficient, or should the doc also roll this much detail
into the migration instructions? We've merged all the changes that avoid
the too-open recommendation, but should we also not consider this bug
closed until the instructions are in the doc itself?

On Wed, Mar 5, 2014 at 2:43 PM, Nathan Kinder <email address hidden> wrote:

> I added some additional content to the "Recommended Actions" section.
> The additional content covers configuring the list address for the
> epehmeral migration ports, the ability to tunnel migration traffic over
> the remote access port, and details about securing the live block
> migration traffic.
>
> I believe this gives a more complete picture of how live migration
> can/should be secured. It also will allow this single OSSN to cover the
> issues described in bug 1240554 instead of having two separte (but
> similar) OSSNs.
>
> The new/reworked sections are below. Feedback would be appreciated.
>
> ----------------------------------------------------------------------
> In addition to requiring authentication for remote access to the libvirt
> daemon
> on your OpenStack Compute nodes, it is also recommended to restrict network
> access such that connectivity is only allowed between the Compute nodes.
>
> The first thing that should be done is to restrict the network interfaces
> that
> the libvirt daemon listens on for remote connections. By default, the
> libvirt
> daemon listens on all interfaces when remote access is enabled. This can
> be
> restricted by setting the following configuration directive in
> /etc/libvirt/libvirtd.conf:
>
> ---- begin example libvirtd.conf snippet ----
> listen_addr = <IP address or hostname>
> ---- end example libvirtd.conf snippet ----
>
> Migration in the libvirt daemon also uses a range of ephemeral ports by
> default. The connections to these ephemeral ports are not authenticated or
> encrypted. It is possible to tunnel the migration traffic over the regular
> libvirt daemon remote access port, which will use the authentication and
> encryption settings that you have defined for that port. It is recommended
> that you do this for the additional security that it provides. To enable
> tunneling of the migration traffic, you must tell your OpenStack Compute
> nodes to set the VIR_MIGRATE_TUNNELLED flag for live migration. This is
> done
> by setting the following directive in /etc/nova/nova.conf:
>
> ---- begin example nova.conf snippet ----
> live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER,
> VIR_MIGRATE_TUNNELLED
> ---- end example nova.conf snippet ----
>
> The tunneling of migration traffic described above does not apply to live
> block
> migration. Live block migration is currently only possible over ephemeral
> ports.
>
> If you choose to use the ephemeral migration ports, there are a few things
> that
> you should be aware of. Unfortunately, there is no way to restrict the
> network
> interfaces that these ephemeral ports will listen on in libvirt versions
> prior
> to 1.1.4. If you are running version 1.1.4 or later of the libvirt
> da...

Read more...

Revision history for this message
Nathan Kinder (nkinder) wrote :

@annegentle
Thanks for you review! I think that the docs should be improved. I had planned to submit a patch for the docs to add a lot of the detail I've written up for the OSSN once this issue is made public.

You've fixed the major issue in the docs to prevent people from newly creating an insecure setup, so that's great for people who have not yet configured live migration. the next step is a OSSN to raise awareness for people who may have already followed the old instructions. These two items should deal with the "security issue" here.

The third step should be to improve the docs to show how you can configure things securely to help people who are setting up live migration instead of requiring them to go hunting for information. It's up to you if want to deal with that in this bug or another bug. The important thing is that this issue be made public and the OSSN is published first.

Revision history for this message
Robert Clark (robert-clark) wrote :

Can we get the OSSN published as soon as possible please?

The anticipation with the security guide was that we'd inter-weave OSSNs throughout, we've done that in a few places but there are also significant gaps. I hope that we can look to include this as a standard part of the process as the new OSSN process rolls out in the future.

Revision history for this message
Nathan Kinder (nkinder) wrote :

@robert-clark
The OSSN is ready to publish if nobody has any further comments about it. I was hoping to get a review of the content by another OSSG member. Does anyone have any last comments before I publish it and make this bug public?

Revision history for this message
Anne Gentle (annegentle) wrote :

+1 to this Robert, thanks.

On Thu, Mar 6, 2014 at 9:56 AM, Robert Clark <email address hidden>wrote:

> Can we get the OSSN published as soon as possible please?
>
> The anticipation with the security guide was that we'd inter-weave OSSNs
> throughout, we've done that in a few places but there are also
> significant gaps. I hope that we can look to include this as a standard
> part of the process as the new OSSN process rolls out in the future.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1287194
>
> Title:
> Docs recommend insecure configuration option
>
> Status in OpenStack Manuals:
> New
> Status in OpenStack Security Advisories:
> Won't Fix
> Status in OpenStack Security Notes:
> In Progress
>
> Bug description:
> From Daniel Berrange:
>
> ---------
> I just came across the following docs for configuring migration with
> nova+libvirt+kvm
>
>
> http://docs.openstack.org/trunk/config-reference/content//section_configuring-compute-migrations.html
>
> http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-migrations.html
>
> At point 7 the docs say
>
> 7. Update the libvirt configurations. Modify the
> /etc/libvirt/libvirtd.conf file:
>
> before : #listen_tls = 0
> after : listen_tls = 0
> before : #listen_tcp = 1
> after : listen_tcp = 1
> add: auth_tcp = "none"
>
> Modify the /etc/init/libvirt-bin.conf file:
>
> before : exec /usr/sbin/libvirtd -d
> after : exec /usr/sbin/libvirtd -d -l
>
> What this does is tell the libvirt daemon listen for client TCP
> connections
> on all network interfaces, and accept any clients performing absolutely
> zero authentication.
>
> Sure this works for migration, but it also allows anyone of the network
> to be able to completely own all your compute hosts, by invoking whatever
> libvirt API calls they like.
>
> This is equivalent to telling someone to configure SSH to allow root
> logins with no passwords or keys at all. Actually it is worse because
> as well as killing authentication, it kills any encryption too.
>
> Libvirt has a choice of 4 secure options for remote access over TCP
>
> - SSH tunnel to libvirtd's UNIX socket
> - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
> - libvirtd TCP socket, with TLS for encryption and x509 client
> certs for authentication
> - libvirtd TCP socket, with TLS for encryption and Kerberos for
> authentication
>
> documenting any of these setups would be better than what's there
> now, which needs to be removed asap.
>
> I don't know whether these docs are the current supported / preferred
> docs for this - they're just what I found via google. If we have other
> docs covering migration setup, they should be checked too.
> ----------------
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-manuals/+bug/1287194/+subscriptions
>

Revision history for this message
Bryan D. Payne (bdpayne) wrote :

I have reviewed the proposed OSSN and it looks good to me. Thanks for the nice writeup Nate!

Revision history for this message
Nathan Kinder (nkinder) wrote :
Download full text (9.5 KiB)

Ok, I'm going to publish the following OSSN and make this bug public:

--------------------------------------------

Live migration instructions recommend unsecured libvirt remote access
---

### Summary ###
When using the KVM hypervisor with libvirt on OpenStack Compute nodes,
live migration of instances from one Compute server to another requires
that the libvirt daemon is configured for remote network connectivity.
The libvirt daemon configuration recommended in the OpenStack
Configuration Reference manual configures libvirtd to listen for
incoming TCP connections on all network interfaces without requiring any
authentication or using any encryption. This insecure configuration
allows for anyone with network access to the libvirt daemon TCP port on
OpenStack Compute nodes to control the hypervisor through the libvirt
API.

### Affected Services / Software ###
Nova, Compute, KVM, libvirt, Grizzly, Havana, Icehouse

### Discussion ###
The default configuration of the libvirt daemon is to not allow remote
access. Live migration of running instances between OpenStack Compute
nodes requires libvirt daemon remote access between OpenStack Compute
nodes.

The libvirt daemon should not be configured to allow unauthenticated
remote access. The libvirt daemon has a choice of 4 secure options for
remote access over TCP. These options are:

 - SSH tunnel to libvirtd's UNIX socket
 - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
 - libvirtd TCP socket, with TLS for encryption and x.509 client
   certificates for authentication
 - libvirtd TCP socket, with TLS for encryption and Kerberos for
   authentication

It is not necessary for the libvirt daemon to listen for remote TCP
connections on all interfaces. Remote network connectivity to the
libvirt daemon should be restricted as much as possible. Remote
access is only needed between the OpenStack Compute nodes, so the
libvirt daemon only needs to listen for remote TCP connections on the
interface that is used for this communication. A firewall can be
configured to lock down access to the TCP port that the libvirt daemon
listens on, but this does not sufficiently protect access to the libvirt
API. Other processes on a remote OpenStack Compute node might have
network access, but should not be authorized to remotely control the
hypervisor on another OpenStack Compute node.

### Recommended Actions ###
If you are using the KVM hypervisor with libvirt on OpenStack Compute
nodes, you should review your libvirt daemon configuration to ensure
that it is not allowing unauthenticated remote access.

Remote access to the libvirt daemon via TCP is configured by the
"listen_tls", "listen_tcp", and "auth_tcp" configuration directives. By
default, these directives are all commented out. This results in remote
access via TCP being disabled.

If you do not need remote libvirt daemon access, you should ensure that
the following configuration directives are set as follows in the
/etc/libvirt/libvirtd.conf configuration file. Commenting out these
directives will have the same effect, as these values match the internal
defaults:

---- begin example libvirtd.conf snippet ----
listen_tls = 1
l...

Read more...

information type: Private Security → Public Security
Revision history for this message
Nathan Kinder (nkinder) wrote :

The OSSN has been published to the following locations:

- Wiki (https://wiki.openstack.org/wiki/OSSN/OSSN-0007)
- <email address hidden>
- <email address hidden>

Changed in ossn:
status: In Progress → Fix Released
Changed in openstack-manuals:
importance: Undecided → Critical
Thierry Carrez (ttx) on 2014-03-19
information type: Public Security → Public
Anne Gentle (annegentle) on 2014-03-21
Changed in openstack-manuals:
status: New → Fix Released
assignee: nobody → Anne Gentle (annegentle)
Anne Gentle (annegentle) on 2014-11-20
Changed in openstack-manuals:
status: Fix Released → In Progress
Revision history for this message
Anne Gentle (annegentle) wrote :

We moved the Configuration Reference away from /trunk/ back in January, but the files were not deleted. I have deleted the files on the FTP server so this page should no longer come up in search results as soon as the next Google index spider does its work.

Changed in openstack-manuals:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-manuals 15.0.0

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers