Host machine exposed to tenant networks via IPv6

[Dustin Lundquist]

When creating a new interface Neutron creates interface and brings link up without disabling default IPv6 binding. By default Linux brings IPv6 link local addresses to all interfaces, this is different behavior than IPv4 where an administrator must explicitly configure an address on the interface.

The is significantly exposed in LinuxBridgeManager ensure_vlan() and ensure_vxlan() where a new VLAN or VXLAN interface is created and set link up before being enslaved in the bridge. In the case of compute node joining and existing network, there is a time window in which VLAN or VXLAN interface is created and has connectivity to the tenant network before it has been enslaved in bridge. Under normal circumstances this time window is less than the time needed to preform IPv6 duplicate address detection, but under high load this assumption may not hold.

I recommend explicitly disabling IPv6 via sysctl on each interface which will be attached to a bridge prior bringing the interface link up. This is already done for the bridge interfaces themselves, but should be done for all Neutron configured interfaces in the default namespace.

This issue was referenced in https://bugs.launchpad.net/neutron/+bug/1459856
Related issue addressed being addressed in Nova: https://review.openstack.org/#/c/198054/

[Dustin Lundquist (dlundquist)]

I have a fix ready for this.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Armando Migliaccio (armando-migliaccio)]

If this is already somewhat public domain, does it make sense to go through this process? Btw, is

https://review.openstack.org/#/c/196199

The fix you are referring to?

[Dustin Lundquist (dlundquist)]

No, https://review.openstack.org/#/c/196199 is not the patch. Attached is patch, unit tests still require modifications.

[Jeremy Stanley (fungi)]

If the issue described here is already considered public knowledge or has been discussed in modest detail in a public venue, then we should switch the report to either Public Security (or just Public it doesn't seem like something for which we'd issue a security advisory). If this issue is not yet publicly known, then the VMT generally leaves it up to the bug reporter and security reviewers for the affected project(s) to determine whether the risk is significant enough to require any embargo period.

[Dustin Lundquist (dlundquist)]

I don't believe the explicit example of the of the LinuxBridge agent's VLAN and VXLAN interface are well known, but generally it is known that IPv6 binds to Neutron/Nova interfaces on the host and there have been a number of cases where IPv6 is exposed to the underlying host. Considering https://bugs.launchpad.net/nova/+bug/1470931 and https://bugs.launchpad.net/neutron/+bug/1302080 I don't think there is much value continue the embargo.

[Dustin Lundquist (dlundquist)]

Updated patch including unit tests.

[Armando Migliaccio (armando-migliaccio)]

Based on the chat I had with Dustin, I'd say we can make this public, as done already.

[Armando Migliaccio (armando-migliaccio)]

@Dustin: please post the patch through the usual review process.

[Jeremy Stanley (fungi)]

I've switched the bug from Private Security to Public Security based on comments 3, 6, 8 and 9 above.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268373
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fc8ebae0351f5b6596951cdfc5cb4259501d84f2
Submitter: Jenkins
Branch: master

commit fc8ebae0351f5b6596951cdfc5cb4259501d84f2
Author: Dustin Lundquist <email address hidden>
Date: Thu Jan 14 23:04:43 2016 -0800

    Prevent binding IPv6 addresses to Neutron interfaces

    Explicitly disable IPv6 on Neutron created interfaces in the default
    namespace before setting link up. Since the default behavior of IPv6 is
    to bind to all interfaces as opposed to IPv4 where an address must be
    explicitly configured we disable IPv6 on each interface before enabling
    the interface. This avoids leaving a time window between when the
    interface is enabled and when it is attached to bridge device during
    which the host could be access from a tenant network.

    Move disable_ipv6() from BridgeDevice to base IPDevice class so it is
    usable by all interfaces. Then we explicitly disable IPv6 on veth
    interfaces in the default namespaces and VXLAN and VLAN interfaces
    created by the LinuxBridge agent.

    In addition vlan interface is moved from LinuxBridgeManager to IPWrapper
    so it can return an IPDevice object.

    Closes-Bug: #1534652
    Change-Id: Id879075f2d5ee42f8ff153e813e7519a4424447b

[YAMAMOTO Takashi (yamamoto)]

MidonetInterfaceDriver in networking-midonet repo has the same issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/287074

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287074
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=6cb84079f9e791b7f2da6e65bc670c008e6f55fd
Submitter: Jenkins
Branch: master

commit 6cb84079f9e791b7f2da6e65bc670c008e6f55fd
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Mar 2 19:02:36 2016 +0900

    Prevent binding IPv6 addresses to Neutron interfaces

    Closes-Bug: #1534652
    Neutron change: Id879075f2d5ee42f8ff153e813e7519a4424447b
    Change-Id: I7ae0d1ad540e81c99681aa00d609f2ad1baa4511

[Miguel Angel Ajo (mangelajo)]

Does this affect kilo/liberty ? I'm adding the potential backport flags for just in case, feel free to remove.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/291275

[Dustin Lundquist (dlundquist)]

Backported to Liberty. This patch would require significant work to backport to Kilo due to changes in LinuxBridge agent.

[Ihar Hrachyshka (ihar-hrachyshka)]

Dustin, do you plan to follow up with Kilo patch?

[Dustin Lundquist (dlundquist)]

Ihar, I could do that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/296659

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/291275
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ccdfd17666698b81074ead89dc865949074295e5
Submitter: Jenkins
Branch: stable/liberty

commit ccdfd17666698b81074ead89dc865949074295e5
Author: Dustin Lundquist <email address hidden>
Date: Thu Jan 14 23:04:43 2016 -0800

    Prevent binding IPv6 addresses to Neutron interfaces

    Explicitly disable IPv6 on Neutron created interfaces in the default
    namespace before setting link up. Since the default behavior of IPv6 is
    to bind to all interfaces as opposed to IPv4 where an address must be
    explicitly configured we disable IPv6 on each interface before enabling
    the interface. This avoids leaving a time window between when the
    interface is enabled and when it is attached to bridge device during
    which the host could be access from a tenant network.

    Move disable_ipv6() from BridgeDevice to base IPDevice class so it is
    usable by all interfaces. Then we explicitly disable IPv6 on veth
    interfaces in the default namespaces and VXLAN and VLAN interfaces
    created by the LinuxBridge agent.

    In addition vlan interface is moved from LinuxBridgeManager to IPWrapper
    so it can return an IPDevice object.

    Conflicts:
     neutron/agent/linux/bridge_lib.py
     neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_linuxbridge_neutron_agent.py

    Closes-Bug: #1534652
    Change-Id: Id879075f2d5ee42f8ff153e813e7519a4424447b
    (cherry picked from commit fc8ebae0351f5b6596951cdfc5cb4259501d84f2)

[Tristan Cacqueray (tristan-cacqueray)]

Is it only the compute host that is exposed before the v[x]lan is enslaved to the bridge ?

If so, then I'd like to remove the OSSA task for the same reason of previous similar bug reports: the compute host needs to be secured properly.

[Dustin Lundquist (dlundquist)]

@Tristan, Any host which is hosting ports for a given Neutron network: usually this would be a compute or network node role. I agree these hosts should be properly secured, but it would be unexpected to an operator for a tenant instance to be connecting to a hypervisor hosts via IPv6 link-local.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296659
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b69bce8a8fe6382549850717be51623239d38769
Submitter: Jenkins
Branch: stable/kilo

commit b69bce8a8fe6382549850717be51623239d38769
Author: Dustin Lundquist <email address hidden>
Date: Thu Jan 14 23:04:43 2016 -0800

    Prevent binding IPv6 addresses to Neutron interfaces

    Explicitly disable IPv6 on Neutron created interfaces in the default
    namespace before setting link up. Since the default behavior of IPv6 is
    to bind to all interfaces as opposed to IPv4 where an address must be
    explicitly configured we disable IPv6 on each interface before enabling
    the interface. This avoids leaving a time window between when the
    interface is enabled and when it is attached to bridge device during
    which the host could be access from a tenant network.

    Move disable_ipv6() from BridgeDevice to base IPDevice class so it is
    usable by all interfaces. Then we explicitly disable IPv6 on veth
    interfaces in the default namespaces and VXLAN and VLAN interfaces
    created by the LinuxBridge agent.

    In addition vlan interface is moved from LinuxBridgeManager to IPWrapper
    so it can return an IPDevice object.

    Conflicts:
     neutron/agent/linux/bridge_lib.py
     neutron/agent/linux/interface.py
     neutron/agent/linux/ip_lib.py
     neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py
     neutron/tests/unit/plugins/ml2/drivers/linuxbridge/agent/test_linuxbridge_neutron_agent.py

    Closes-Bug: #1534652
    Change-Id: Id879075f2d5ee42f8ff153e813e7519a4424447b
    (cherry picked from commit fc8ebae0351f5b6596951cdfc5cb4259501d84f2)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/311573

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tristan Cacqueray (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/311573
Reason: Oups, mea-culpa... I didn't realised master change was merged before stable/mitaka. This backport is void.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 2015.1.4 release.

[Tristan Cacqueray (tristan-cacqueray)]

Based on a similar report (bug 1302080), I've closed the OSSA task. However I've added an OSSN task to discuss an eventual Note about compute and controller firewalling requirements.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 7.1.0 release.

[Luke Hinds (lhinds)]

Hmm ok, I was already working on this. Did you want to take over it?

[Vinay Potluri (vinay-potluri)]

Luke Hinds,
I'm new to OpenSource and started working on it at the midcycle. If you have already worked on it we can combine our work on the bug.

[Luke Hinds (lhinds)]

Vinay, no problem. Can I share a doc with your gmail account? I can send you what I have researched, and support you in submitting a patch and getting a note published.

Its always good to have new folk come on to help :)

[Vinay Potluri (vinay-potluri)]

Sure Luke, please share it with me. It will be very helpful.
On Aug 17, 2016 5:26 PM, "Luke Hinds" <email address hidden> wrote:

> Vinay, no problem. Can I share a doc with your gmail account? I can send
> you what I have researched, and support you in submitting a patch and
> getting a note published.
>
> Its always good to have new folk come on to help :)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1534652
>
> Title:
> Host machine exposed to tenant networks via IPv6
>
> Status in networking-midonet:
> Fix Released
> Status in neutron:
> Fix Released
> Status in neutron kilo series:
> New
> Status in OpenStack Security Advisory:
> Won't Fix
> Status in OpenStack Security Notes:
> Confirmed
>
> Bug description:
> When creating a new interface Neutron creates interface and brings
> link up without disabling default IPv6 binding. By default Linux
> brings IPv6 link local addresses to all interfaces, this is different
> behavior than IPv4 where an administrator must explicitly configure an
> address on the interface.
>
> The is significantly exposed in LinuxBridgeManager ensure_vlan() and
> ensure_vxlan() where a new VLAN or VXLAN interface is created and set
> link up before being enslaved in the bridge. In the case of compute
> node joining and existing network, there is a time window in which
> VLAN or VXLAN interface is created and has connectivity to the tenant
> network before it has been enslaved in bridge. Under normal
> circumstances this time window is less than the time needed to preform
> IPv6 duplicate address detection, but under high load this assumption
> may not hold.
>
> I recommend explicitly disabling IPv6 via sysctl on each interface
> which will be attached to a bridge prior bringing the interface link
> up. This is already done for the bridge interfaces themselves, but
> should be done for all Neutron configured interfaces in the default
> namespace.
>
> This issue was referenced in https://bugs.launchpad.net/
> neutron/+bug/1459856
> Related issue addressed being addressed in Nova:
> https://review.openstack.org/#/c/198054/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/networking-midonet/+bug/1534652/+subscriptions
>

[Luke Hinds (lhinds)]

If there are any core Neutron reviewers, could you please look at the review[1]. We need a +1 before we can send out the security note.

[1] https://bugs.launchpad.net/ossn/+bug/1534652

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 2015.1.4 release.