Small window of access to kvmhost with l3 driver and floating ips

[Mingyan Bao]

We noticed that when floating IP addresses were being assigned to instances, the L3 driver binds the IP to the kvmhost’s public interface first, and then setup the NAT rules. This gives a window of opportunity for traffic for floating IP to go to the kvmhost instead of the instance. I traced the issue to nova/network/l3.py, add_floating_ip method:

Original:
    def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
        linux_net.bind_floating_ip(floating_ip, l3_interface_id)
        linux_net.ensure_floating_forward(floating_ip, fixed_ip)

Modified:
    def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
        linux_net.ensure_floating_forward(floating_ip, fixed_ip)
        linux_net.bind_floating_ip(floating_ip, l3_interface_id)

I think setting up the NAT rules first then binding the IP should be okay. I tested and it seems to work.

[Vish Ishaya (vishvananda)]

makes sense to do them in the reverse order. I would probably be ok making this bug public since it the window is incredibly small and there is only an exploit if there is a vulnerable service running on the host

[Thierry Carrez (ttx)]

Yes, this is a welcome strengthening (and should definitely be fixed) but there is no real vulnerability to be exploited, so we won't embargo the fix and do an advisory about it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/21228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21228
Committed: http://github.com/openstack/nova/commit/12c73ee65372fd1b0a3d3417cd91e548fd7afd2b
Submitter: Jenkins
Branch: master

commit 12c73ee65372fd1b0a3d3417cd91e548fd7afd2b
Author: mingyan bao <email address hidden>
Date: Mon Feb 4 15:52:33 2013 -0700

    l3.py,add_floating_ip: setup NAT before binding

    fix for bug 1100435

    Change-Id: Iad022f61297fe26edb230ba7b9e31d73df99b5a5

[Robert Clark (robert-clark)]

Glad to see this is fixed. I would consider it to be somewhat concerning for those of us running a public cloud, sure there's a small window but it's repeatable, I believe a motivated attacker with other points-of-presence on-cloud could probably find interesting ways to abuse this.

All that being said, it's far from the lowest hanging fruit.

[Jason Hullinger (jason-hullinger)]

That's why it takes a few seconds from when a VM launches to when you can
actually ssh to it.

On 4/4/13 10:53 AM, "Robert Clark" <email address hidden> wrote:

>Glad to see this is fixed. I would consider it to be somewhat concerning
>for those of us running a public cloud, sure there's a small window but
>it's repeatable, I believe a motivated attacker with other points-of-
>presence on-cloud could probably find interesting ways to abuse this.
>
>All that being said, it's far from the lowest hanging fruit.
>
>--
>You received this bug notification because you are a member of OpenStack
>Security Group, which is subscribed to OpenStack.
>https://bugs.launchpad.net/bugs/1100435
>
>Title:
> Small window of access to kvmhost with l3 driver and floating ips
>
>Status in OpenStack Compute (Nova):
> Fix Released
>
>Bug description:
> We noticed that when floating IP addresses were being assigned to
> instances, the L3 driver binds the IP to the kvmhost¹s public
> interface first, and then setup the NAT rules. This gives a window of
> opportunity for traffic for floating IP to go to the kvmhost instead
> of the instance. I traced the issue to nova/network/l3.py,
> add_floating_ip method:
>
> Original:
> def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
> linux_net.bind_floating_ip(floating_ip, l3_interface_id)
> linux_net.ensure_floating_forward(floating_ip, fixed_ip)
>
> Modified:
> def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
> linux_net.ensure_floating_forward(floating_ip, fixed_ip)
> linux_net.bind_floating_ip(floating_ip, l3_interface_id)
>
>
> I think setting up the NAT rules first then binding the IP should be
>okay. I tested and it seems to work.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/nova/+bug/1100435/+subscriptions

[Kurt Seifried (kseifried)]

A few seconds is a long time when it comes to network traffic/talking to hosts, this exposes the underlying system in a way that is probably not meant to happen. I think this probably deserves a CVE, is there any reason this has not gotten one?

[Robert Clark (robert-clark)]

+1 for a CVE

[Randy Perryman (randy-perryman)]

+1

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Robert Clark
Sent: Thursday, April 04, 2013 3:24 PM
To: Perryman, Randy
Subject: [Bug 1100435] Re: Small window of access to kvmhost with l3 driver and floating ips

+1 for a CVE

--
You received this bug notification because you are a member of OpenStack Security Group, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1100435

Title:
  Small window of access to kvmhost with l3 driver and floating ips

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  We noticed that when floating IP addresses were being assigned to
  instances, the L3 driver binds the IP to the kvmhost’s public
  interface first, and then setup the NAT rules. This gives a window of
  opportunity for traffic for floating IP to go to the kvmhost instead
  of the instance. I traced the issue to nova/network/l3.py,
  add_floating_ip method:

  Original:
      def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
          linux_net.bind_floating_ip(floating_ip, l3_interface_id)
          linux_net.ensure_floating_forward(floating_ip, fixed_ip)

  Modified:
      def add_floating_ip(self, floating_ip, fixed_ip, l3_interface_id):
          linux_net.ensure_floating_forward(floating_ip, fixed_ip)
          linux_net.bind_floating_ip(floating_ip, l3_interface_id)

  I think setting up the NAT rules first then binding the IP should be okay. I tested and it seems to work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1100435/+subscriptions

[Thomas Biege (thomas-suse-deactivatedaccount)]

Yes, this needs a CVE-ID. (BTW most of the other issue I saw getting fixed in Grizzly within this week need one too from my POV).