quantum-ns-metadata-proxy listens on external interfaces too

Bug #1187102 reported by Darragh O'Reilly
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Invalid
Undecided
Unassigned
neutron
Fix Released
Medium
Cedric Brandily
Juno
Fix Released
Undecided
Unassigned

Bug Description

Running Grizzy 2013.1 on Ubuntu 12.04. Three nodes: controller, network and compute.

netnode# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 18462/python

So this router is uplinked to an external network:

netnode# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 ip -4 a
14: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
23: qr-123f9b7f-43: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 172.17.17.1/24 brd 172.17.17.255 scope global qr-123f9b7f-43
24: qg-c8a6a6cd-6d: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 192.168.101.2/24 brd 192.168.101.255 scope global qg-c8a6a6cd-6d

Now from outside can do:

$ nmap 192.168.101.2 -p 9697
Starting Nmap 6.00 ( http://nmap.org ) at 2013-06-03 13:45 IST
Nmap scan report for 192.168.101.2
Host is up (0.0018s latency).
PORT STATE SERVICE
9697/tcp open unknown

As a test I tried changing namespace_proxy.py so it would not bind to 0.0.0.0

    proxy.start(handler, self.port, host='127.0.0.1')

but the metadata stopped working. In iptables this rule is being hit:

  -A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697

I'm guessing the intention of that rule is also change the destination address to 127.0.0.1 ? as there is this:

  -A quantum-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT

but the counters show that this rule is not being hit. Anyway the default policy for INPUT is ACCEPT.

From the iptables man page:
  REDIRECT
       "... It redirects the packet to the machine itself by changing the destination IP to the primary address of the incoming interface
       (locally-generated packets are mapped to the 127.0.0.1 address). ..."

so the primary address of the incoming interface is 172.17.17.1, not 127.0.0.1.

So I manually deleted the "-j REDIRECT --to-ports 9697" and added "-j DNAT --to-destination 127.0.0.1:9697" but that didn't work - seems like it is not possible: http://serverfault.com/questions/351816/dnat-to-127-0-0-1-with-iptables-destination-access-control-for-transparent-soc

So I tried changing the ns proxy to listen on 172.17.17.1. I think this is the one and only address it should bind to anyway.

    proxy.start(handler, self.port, host='172.17.17.1') # hardwire as a test

Stopped the l3-agent, killed the quantum-ns-metadata-proxy and restarted the l3-agent. But the ns proxy gave an error:

Stderr: 'cat: /proc/10850/cmdline: No such file or directory\n'
2013-06-03 15:05:18 ERROR [quantum.wsgi] Unable to listen on 172.17.17.1:9697
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 72, in start
    backlog=backlog)
  File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
    sock.bind(addr)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 99] Cannot assign requested address

The l3-agent.log shows the agent deleted the port qr-123f9b7f-43 at 15:05:10 and did not recreate it until 15:05:19 - ie a second too late for the ns proxy. From looking at the code it seems the l3-agent spawns the ns proxy just before it plugs its ports. I was able to start the ns proxy manually with the command line from the l3-agent log, and the metadata worked and was not reachable from outside.

summary: - quantum-ns-proxy listens on external interfaces too
+ quantum-ns-metadata-proxy listens on external interfaces too
description: updated
Revision history for this message
Thierry Carrez (ttx) wrote :

Adding Quantum PTL for debunking whether this constitues an exploitable vulnerability.
I suppose information can be leaked from that metadata proxy to the external network ?

Changed in ossa:
status: New → Incomplete
Revision history for this message
Darragh O'Reilly (darragh-oreilly) wrote :

Apart from DoS, I can't think of a particular exploit - but I'm not a security expert. Source IP addresses can be spoofed, but the Quantum router will route replies to the internal fixed networks or drop if they don't exist. I'm just pointing out that the proxy has no reason to listen on the external interface and therefore it shouldn't.

A quick fix might be to block that port:
-A quantum-l3-agent-INPUT -d 192.168.101.2/32 -i qg-c8a6a6cd-6d -p tcp -m tcp --dport 9697 -j DROP

or just block all tcp - no local processes need to receive traffic from it:
-A quantum-l3-agent-INPUT -i qg-c8a6a6cd-6d -p tcp -j DROP

or block everything (including ICMP)
-A quantum-l3-agent-INPUT -i qg-c8a6a6cd-6d -j DROP

Revision history for this message
Thierry Carrez (ttx) wrote :

Yeah, that should definitely be fixed, but i see no reason to embargo the fix or restrict access to this bug.
Waiting for confirmation from other Quantum core folks before opening up.

Revision history for this message
Mark McClain (markmcclain) wrote :

Need to a test a bit a more, but I'm not sure this an exploitable vulnerability.

Revision history for this message
Thierry Carrez (ttx) wrote :

@MarkMcClain: any more input ? OK to open ?

Thierry Carrez (ttx)
Changed in ossa:
status: Incomplete → Invalid
information type: Private Security → Public
tags: added: metadata
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Let's consider if this should be fixed at all

Changed in neutron:
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
Mark McClain (markmcclain) wrote :

moving back to new.. because more discussion needed

Revision history for this message
Cedric Brandily (cbrandily) wrote :

Confirmed on icehouse and juno

Changed in neutron:
status: Incomplete → Confirmed
assignee: nobody → Cedric Brandily (cbrandily)
Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/133484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/133506

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/143411

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/135203
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c7e533c3679a1f4a612f3b53354cb7cb5bc1ba12
Submitter: Jenkins
Branch: master

commit c7e533c3679a1f4a612f3b53354cb7cb5bc1ba12
Author: Cedric Brandily <email address hidden>
Date: Tue Nov 18 10:34:30 2014 +0100

    Allow IptablesManager to manage mangle table

    This change enables the IptablesManager to manage mangle table (used
    by daughter change).

    Partial-Bug: #1187102
    Change-Id: Ic2d681f1515aaa541c6d137ce981622f2fff90e5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/133506
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/133484
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1d776bc16c033f33e61fd6832f2e94e24cdd1c5f
Submitter: Jenkins
Branch: master

commit 1d776bc16c033f33e61fd6832f2e94e24cdd1c5f
Author: Cedric Brandily <email address hidden>
Date: Mon Nov 10 14:46:51 2014 +0100

    Allow to request metadata proxy only with redirection

    metadata service should be requested on 169.254.169.254:80 and router
    namespace iptables rules redirect the request to the metadata-ns-proxy
    on 127.0.0.1:$metadata_port. But currently the metadata-ns-proxy can be
    requested directly on $router-ip:$metadata_port.

    To avoid such behavior, this change marks packets redirection in mangle
    table (PREROUTING), redirects (PREROUTING) them in nat table, accepts
    them in filter table (INPUT) using the mark. Packets send to the
    metadata proxy port without mark (so directly) are dropped. The
    mark can be configured through the new option metadata_access_mark.

    Remark: redirected packets are not local packets (in general), so
    setting metadata proxy server host to 127.0.0.1 will disallow direct
    queries but so redirected queries.

    DocImpact
    Partial-Bug: #1187102
    Change-Id: I6a9bb12c8bf68c6fcf4e4060f8dfe44a309a41da

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/143411
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/133506
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b049971c5652adc8e6146f15180ceccc58f8ae9a
Submitter: Jenkins
Branch: master

commit b049971c5652adc8e6146f15180ceccc58f8ae9a
Author: Assaf Muller <email address hidden>
Date: Mon Dec 22 17:01:37 2014 +0200

    Allow to request metadata proxy only from internal interfaces

    Currently the metadata service can be requested on 169.254.169.254:80
    from all interfaces including external interfaces. This change updates
    PREROUTING rules to allow request on 169.254.169.254:80 only from
    internal interfaces.

    Change-Id: I44a9e03992f9e2a7bd4d798ae69d8aa7d75d3078
    Closes-Bug: #1187102

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → kilo-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-rc1 → 2015.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/199195

Changed in neutron:
importance: Low → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/199195
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f548775c05f282204f5eafbd21ea12a9db7f1946
Submitter: Jenkins
Branch: stable/juno

commit f548775c05f282204f5eafbd21ea12a9db7f1946
Author: Assaf Muller <email address hidden>
Date: Mon Dec 22 17:01:37 2014 +0200

    Allow to request metadata proxy only from internal interfaces

    Currently the metadata service can be requested on 169.254.169.254:80
    from all interfaces including external interfaces. This change updates
    PREROUTING rules to allow request on 169.254.169.254:80 only from
    internal interfaces.

    L3 agent code has been refactored in Kilo which explains differences
    between the original change and the cherrypick.

    Closes-Bug: #1187102
    (cherry picked from commit b049971c5652adc8e6146f15180ceccc58f8ae9a)

    Conflicts:
     neutron/agent/metadata/driver.py
     neutron/tests/unit/agent/metadata/test_driver.py

    Change-Id: I44a9e03992f9e2a7bd4d798ae69d8aa7d75d3078

tags: added: in-stable-juno
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/207168

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/207169

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/207168
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=580f6c6d7a098011949e8a705791ff39842b1630
Submitter: Jenkins
Branch: stable/juno

commit 580f6c6d7a098011949e8a705791ff39842b1630
Author: Cedric Brandily <email address hidden>
Date: Tue Nov 18 10:34:30 2014 +0100

    Allow IptablesManager to manage mangle table

    This change enables the IptablesManager to manage mangle table (used
    by daughter change).

    It's a security backport.

    Partial-Bug: #1187102
    Change-Id: Ic2d681f1515aaa541c6d137ce981622f2fff90e5
    (cherry picked from commit c7e533c3679a1f4a612f3b53354cb7cb5bc1ba12)
    Conflicts:
     neutron/tests/functional/agent/linux/base.py
     neutron/tests/unit/test_iptables_manager.py

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/207169
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=15cb18bebc1efaccbf122367e1a7b552c53192cd
Submitter: Jenkins
Branch: stable/juno

commit 15cb18bebc1efaccbf122367e1a7b552c53192cd
Author: Cedric Brandily <email address hidden>
Date: Mon Nov 10 14:46:51 2014 +0100

    Allow to request metadata proxy only with redirection

    metadata service should be requested on 169.254.169.254:80 and router
    namespace iptables rules redirect the request to the metadata-ns-proxy
    on 127.0.0.1:$metadata_port. But currently the metadata-ns-proxy can be
    requested directly on $router-ip:$metadata_port.

    To avoid such behavior, this change marks packets redirection in mangle
    table (PREROUTING), redirects (PREROUTING) them in nat table, accepts
    them in filter table (INPUT) using the mark. Packets send to the
    metadata proxy port without mark (so directly) are dropped. The
    mark can be configured through the new option metadata_access_mark.

    Remark: redirected packets are not local packets (in general), so
    setting metadata proxy server host to 127.0.0.1 will disallow direct
    queries but so redirected queries.

    It's a security backport.

    DocImpact
    Partial-Bug: #1187102
    Change-Id: I6a9bb12c8bf68c6fcf4e4060f8dfe44a309a41da
    (cherry picked from commit 1d776bc16c033f33e61fd6832f2e94e24cdd1c5f)
    Conflicts:
     neutron/agent/l3/config.py
     neutron/agent/metadata/driver.py
     neutron/tests/unit/agent/metadata/test_driver.py

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.