Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4 Floating IPs

Bug #1322945 reported by Thiago Martins
260
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Confirmed
Undecided
Unassigned
neutron
Confirmed
Critical
Aaron Rosen

Bug Description

Hey Stackers!

I think I found a serious BUG in Neutron...

When a regular user from random a OpenStack Project, creates a static IPv6 subnet, from within its own account, and attach it to its L3 Router, then, it breaks the IPv4 Floating IPs at the Network Node for an entire OpenStack installation (Region / AZ), making it unusable, plus, that External Network (that one where the Floating IPs come from) breaks too, it becomes impossible to delete it.

Here is how I'm reproducing it:

1- Install OpenStack IceHouse with Ubuntu 14.04, using the topology "Per-Project Router with Private Networks";

2- Logged with "admin" user, create a Shared Public IPv4 External Network as usual (http://docs.openstack.org/icehouse/install-guide/install/apt/content/neutron_initial-external-network.html);

3- Logged as a Project (#1) regular user (i.e. not admin), create its Network as usual, Router attached to the shared External Network, private IPv4 subnet attached to its Router. Example: http://i.imgur.com/lauqDUQ.png ;

4- Still logged as a regular user at Project #1, creates an IPv6 subnet to its private net (like 2008:129:2bf:cd1::/64), with DHCP disabled, no DNS Servers (example: neutron subnet-create --ip-version 6 --disable-dhcp --tenant-id $PROJECT1_TENANT_ID privatenet1 2008:129:2bf:cd1::/64) - Example: http://i.imgur.com/tPQX21E.png ;

5- Allocate a couple Floating IPs for Project #1;

6- Create a Instance and attach a Floating IPv4 to it, test it (it works as expected). Working Example: http://i.imgur.com/xE5brug.png ;

7- Attach the "Project Router" to the tenant IPv6 private subnet (source of the BUG?) = http://i.imgur.com/WfTzpOl.png & http://i.imgur.com/sRAhA6P.png ;

8- Create a new ("dual-stacked") Instance;

9- Here comes the BUG side effects: try to attach an IPv4 at this "dual-stacked" Instance (using IPv4 ports, of course), it will not work! The Floating IPv4 seems to be attached but, the IP does not appear at its L3 Router (Network Node Project Namespace Router) as expected!

9.1- The IPv4 Floating IP 172.20.0.6 (attached after adding "Router #1" to the IPv6 subnet), did not appeared at the L3 Router. Details: http://i.imgur.com/QFSOa3T.png & http://paste.openstack.org/show/81401/ ;

From this point, the Floating IPs from External Network stops working for the entire cloud (Region / AZ)!

Also, it is impossible to clean it up, by deleting the external network, it will not work, triggering the following error: http://paste.openstack.org/show/81402/

I know that IPv6 isn't officially supported by OpenStack but, my point is that a regular user can literally breaks the L3 Router by simple adding a innocent IPv6 subnet to its Router, affecting all other tenants (it effectively breaks the External Network that belongs to the "admin" user). So, because of this, I'm also marking this as a security vulnerability.

Regards,
Thiago

Thiago Martins (martinx)
information type: Private Security → Public
Revision history for this message
Jeremy Stanley (fungi) wrote :

Since you mention this may be a security vulnerability (potential denial of service attack) in a supported release, I've switched the bug from public to public security and added an OSSA task in case it warrants an advisory.

information type: Public → Public Security
Revision history for this message
Thiago Martins (martinx) wrote : Re: [Bug 1322945] Re: Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4 Floating IPs

Okay, no problem... I've marked it as a security issue mostly because a
regular OpenStack user can break the Network Node and affect everybody else
on the same node / region / az... So, it is a big risk for Public Cloud
providers based on IceHouse using this topology (Neutron L3 / GRE / VXLAN).

The "VLAN Provider Networks" (or Flat Networks) aren't affected.

Best,
Thiago

On 26 May 2014 15:19, Jeremy Stanley <email address hidden> wrote:

> Since you mention this may be a security vulnerability (potential denial
> of service attack) in a supported release, I've switched the bug from
> public to public security and added an OSSA task in case it warrants an
> advisory.
>
> ** Information type changed from Public to Public Security
>
> ** Also affects: ossa
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1322945
>
> Title:
> Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4
> Floating IPs
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1322945/+subscriptions
>

tags: added: ipv6
tags: added: l3-ipam-dhc
Changed in neutron:
importance: Undecided → High
tags: added: icehouse-backport-potential
Revision history for this message
Aaron Rosen (arosen) wrote :
Download full text (4.8 KiB)

I was able to reproduce this issue:

arosen@arosen-MacBookPro:~/devstack$ neutron subnet-create --ip-version 6 --disable-dhcp 556f3c44-01bb-468a-8064-10c833eb3715 2008:129:2bf:cd1::/64^C
arosen@arosen-MacBookPro:~/devstack$ neutron subnet-list
+--------------------------------------+----------------+-----------------------+---------------------------------------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+-----------------------+---------------------------------------------------------------------------------+
| 556f3c44-01bb-468a-8064-10c833eb3715 | private-subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| e8180cc2-e5ea-4892-997e-e9ff96b426f4 | | 2008:129:2bf:cd1::/64 | {"start": "2008:129:2bf:cd1::2", "end": "2008:129:2bf:cd1:ffff:ffff:ffff:fffe"} |
+--------------------------------------+----------------+-----------------------+---------------------------------------------------------------------------------+
arosen@arosen-MacBookPro:~/devstack$ neutron router-interface-add ^C
arosen@arosen-MacBookPro:~/devstack$ neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 96d4a3cb-4138-41af-b88e-8ebe5405f2bb | router1 | {"network_id": "b9869a8e-cc7b-409f-99c9-ae1a2a4a4855", "enable_snat": true} |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
arosen@arosen-MacBookPro:~/devstack$ neutron router-interface-add router1 e8180cc2-e5ea-4892-997e-e9ff96b426f4
Added interface c2466710-5c40-41a8-9c46-3933134a0785 to router router1.
arosen@arosen-MacBookPro:~/devstack$ ping 172.24.4.4
PING 172.24.4.4 (172.24.4.4) 56(84) bytes of data.
64 bytes from 172.24.4.4: icmp_seq=1 ttl=63 time=0.812 ms
64 bytes from 172.24.4.4: icmp_seq=2 ttl=63 time=0.230 ms
64 bytes from 172.24.4.4: icmp_seq=3 ttl=63 time=0.575 ms
64 bytes from 172.24.4.4: icmp_seq=4 ttl=63 time=0.507 ms
64 bytes from 172.24.4.4: icmp_seq=5 ttl=63 time=0.564 ms
^C
--- 172.24.4.4 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3997ms
rtt min/avg/max/mdev = 0.230/0.537/0.812/0.187 ms
arosen@arosen-MacBookPro:~/devstack$ neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 55dab106-8d5a-4b54-93bf-b4a91b5fbaec | 10.0....

Read more...

Changed in neutron:
status: New → Confirmed
importance: High → Critical
Revision history for this message
Aaron Rosen (arosen) wrote :

The issue seems to be due to how the subnet mask format is passed in. This is a security issue for all plugins using the l3-agent as it causes the l3-agent to no longer forward traffic. In the future please file these type of bugs as private..

Changed in neutron:
assignee: nobody → Aaron Rosen (arosen)
Changed in ossa:
status: New → Confirmed
Revision history for this message
Aaron Rosen (arosen) wrote :

Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-96d4a3cb-4138-41af-b88e-8ebe5405f2bb', 'iptables-restore', '-c']
Exit code: 2
Stdout: ''
Stderr: "iptables-restore v1.4.18: invalid mask `64' specified\nError occurred at line: 50\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n" from (pid=15851) execute /opt/stack/neutron/neutron/agent/linux/utils.py:74

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/95902

Revision history for this message
Thiago Martins (martinx) wrote : Re: [Bug 1322945] Re: Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4 Floating IPs

Okay, sorry... Next time I'll make it private only... My bad...

On 27 May 2014 17:17, Aaron Rosen <email address hidden> wrote:

> The issue seems to be due to how the subnet mask format is passed in.
> This is a security issue for all plugins using the l3-agent as it causes
> the l3-agent to no longer forward traffic. In the future please file
> these type of bugs as private..
>
> ** Changed in: neutron
> Assignee: (unassigned) => Aaron Rosen (arosen)
>
> ** Changed in: ossa
> Status: New => Confirmed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1322945
>
> Title:
> Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4
> Floating IPs
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1322945/+subscriptions
>

Thiago Martins (martinx)
information type: Public Security → Private Security
Revision history for this message
Jeremy Stanley (fungi) wrote :

There is no real benefit to switching this particular bug to private at this point, since it has already been public for several days and copies of it are now on mailing list archives and in other places indexed by Web search engines. Aaron was merely trying to point out that you have the option of keeping future bugs of this nature private to allow the developers some time to work on resolving them prior to publication.

information type: Private Security → Public Security
Revision history for this message
Thiago Martins (martinx) wrote :

Okay Jeremy, got it... Sorry for that... :-)

On 28 May 2014 11:56, Jeremy Stanley <email address hidden> wrote:

> *** This bug is a duplicate of bug 1309195 ***
> https://bugs.launchpad.net/bugs/1309195
>
> There is no real benefit to switching this particular bug to private at
> this point, since it has already been public for several days and copies
> of it are now on mailing list archives and in other places indexed by
> Web search engines. Aaron was merely trying to point out that you have
> the option of keeping future bugs of this nature private to allow the
> developers some time to work on resolving them prior to publication.
>
> ** Information type changed from Private Security to Public Security
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1322945
>
> Title:
> Attaching a IPv6 private subnet to a L3 Router, breaks it and its IPv4
> Floating IPs
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1322945/+subscriptions
>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/95902
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=77046d0d889a8bff095083ae80a2accd0e25406b
Submitter: Jenkins
Branch: master

commit 77046d0d889a8bff095083ae80a2accd0e25406b
Author: Aaron Rosen <email address hidden>
Date: Tue May 27 13:39:39 2014 -0700

    Make linux.utils.execute log error on return codes

    Previously, the execute method in neutron logs everything as debug which hides
    a lot of extremely fatal errors like unable to apply security group rules!
    This patch changes this code so that we log all non 0 returns as error.

    Change-Id: I7328e62269212ccd9c7950ff064a3e337de56918
    Closes-bug: 1323832
    Related-bug: 1322945

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/icehouse)

Related fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/97117

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/havana)

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/97118

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/icehouse)

Change abandoned by Aaron Rosen (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/97117

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/havana)

Change abandoned by Aaron Rosen (<email address hidden>) on branch: stable/havana
Review: https://review.openstack.org/97118

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.