VXLAN Overlay ping issue when Gateway IP is set to one of local NIC's IP address

[Uday]

There's an issue when a VXLAN overlay VM tries to ping an overlay IP address that is also the same as one of the host machine's local IP addresses. In my setup, I've tried pinging the overlay VM's router's IP address. Here are the details:

VXLAN Id is 100 (this number is immaterial, what matters is that we use VXLAN for tenant traffic)

Overlay VM:
IP: 10.0.1.3/24
GW: 10.0.1.1

Host Info:
enp21s0f0: 1.1.1.5/24 (This interface is used to contact the controller as well as for encapsulated datapath traffic.

qbr89a962f7-9b: Linux Bridge to which the Overlay VM connects. No IP address on this one.

brctl show:
qbr89a962f7-9b 8000.56f6fefb9d5c no qvb89a962f7-9b
                                                        tap89a962f7-9b

ifconfig qbr89a962f7-9b
qbr89a962f7-9b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::54f6:feff:fefb:9d5c prefixlen 64 scopeid 0x20<link>
        ether 56:f6:fe:fb:9d:5c txqueuelen 0 (Ethernet)
        RX packets 916 bytes 27072 (26.4 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 10 bytes 780 (780.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I am using a previously unused NIC named eno1 for this example. When eno1 has no IP address, ping from the overlay VM to the router is successful. ARP on the VM shows the correct MAC resolution. When I set eno1 to 10.0.1.1, ARP on the overlay VM show's qbr89a962f7-9b's MAC address and ping never succeeds.

When things work OK ARP for 10.0.1.1 is fa:16:3e:0c:52:6d

When eno1 is set to 10.0.1.1 ARP resolution is incorrect, 10.0.1.1 resolves to 56:f6:fe:fb:9d:5c and ping never succeeds. I've deleted ARPs to ensure that resolution is triggered. It appears as of the OVS br-int never received the ARP request.

Thanks,
-Uday

[Tristan Cacqueray (tristan-cacqueray)]

Hi, is there a reason why you reported this bug as a security vulnerability ?

[Jeremy Stanley (fungi)]

This looks like it was intended to be a normal bug report, but you've instead classified it as a potential security vulnerability to be kept secret. If you did not intend for this to be a "private security" bug, please switch it to "public" instead. If this is actually describing a suspected vulnerability, please adjust the bug description or add a comment to explain the associated security risks implied by this bug.

[Uday (nagaraj)]

I marked this as a security issue since:
- It exposes elements of the host node to the overlay clients, which could be misused.
- This can cause overlay clients to become unreachable leading to service outages (Think if an admin adds an IP to a host node after Overlay clients have been set up)
- The issue is happening due to the use of Linux bridges to implement security groups.

If you disagree with these points, I can switch the classification to public.

Thanks,
-Uday

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Uday (nagaraj)]

If there is no activity on this bug by the end of the day, can I mark this issue as public? Asking since there does not seem to be any confirmation whether this is a real security issue or not.

Thanks,
-Uday

[Jeremy Stanley (fungi)]

We're hoping one of the Neutron developers will have a chance to confirm whether this represents an exploitable risk sufficient to require discussing/reviewing proposed solutions in private.

[Kyle Mestery (mestery)]

To understand this issue, what seems to be happening is that you're configuring a local interface on the host with the same IP as the default gw for your overlay network. In that case, the host IP stack will (correctly) route traffic to the GW, which it thinks is local. I have a couple of concerns with this:

1. You need physical access to the host to do this.
2. I suspect this also is a problem with GRE tunnels, and in turn, OVS since OVS supports GRE and Linuxbridge doesn't.

I'm not sure this is an actual security advisory however, since it does require access to the underlying host itself, at which point there are many other things to do which are bad than just taking over an overlay network GW IP address.

[Uday (nagaraj)]

Hi Kyle,

Thanks for looking at this.

Firstly, the issue is not restricted to arping the default GW of the overlay VMs. This issue will happen when we try to resolve any IP address that also happens to be configured on the local host. So the scope of the issue is much larger than only the IP address of the default GW.

Second, this issue happens when the IP subnet of the infrastructure components is the same as that of the overlay, and this is the fundamental flaw here, since overlay networks were supposed to be completely independent of underlay networks. (e.g. in the case of bring-your-own-ip, esp. where the provider and tenant both want to use the same private IP address ranges)

Thirdly, I dont think any prior knowledge of the physical infrastructure is needed, it is a chance factor which gets magnified by the use of private address ranges and BYOP kind of deployments.

Lastly, you are right about the issue being present in GRE deployments also, since this issue happens before the packets get to any point that needs encapsulation.

In conclusion, I agree that this may not be a serious security flaw, but it has the potential to be disruptive without either the tenant or the provider being aware of the root cause if the setup is as explained earlier.

Please let me know how I can get this issue a wider coverage so it can be viewed/worked on by someone.

Thanks,
-Uday

[Kyle Mestery (mestery)]

I'd really like to get Assaf included here and/or Carl and get their opinions. Jeremy, what is the procedure for adding folks who are not normally security group members of Neutron? Can I add them and get them involved?

[Tristan Cacqueray (tristan-cacqueray)]

Kyle, feel free to add anyone able to help here using the "Subscribe someone else" button

[Kyle Mestery (mestery)]

Added Carl Baldwin to get his inputs here as well.

[Carl Baldwin (carl-baldwin)]

I'm not clear on how enol fits in to the picture. If enol were a subordinate to qbr89a962f7-9b then I might expect something like this. Except the mac address would've been that of enol and not qbr89a962f7-9b. I assume it is not subordinate (it shouldn't be).

I tried a small experiment in a devstack I had lying around. I created a subnet with the same subnet as the host network: 10.224.36.0/24 on eth0. I made the gateway of the subnet the same as the host's IP: 10.224.36.34. Given what I understand of this bug, I expected this to exhibit the problem. Quoting Uday: "this issue happens when the IP subnet of the infrastructure components is the same as that of the overlay." However, it worked fine in my case.

Is there anything else that might be misconfigured on the system in question? Could you provide full output from at least "brctl show", "ip addr show", and "ip route" when the system is in the state where arp is getting the wrong answer?

As for being a security vulnerability, I'm not sure. Arp seems to be misbehaving but there isn't an indication that IP traffic gets redirected, is there?

[Uday (nagaraj)]

Hi Carl,

eno1 was just taken as an example since I did not want to mess up my tunnel endpoint IP addresses. I just wanted to show the behavior when a NIC on my host has the same IP as an overlay IP address.

We even reproduced this issue in another setup, just to confirm the issue and observed the same behavior.

Could it be that your devstack setup was an all-in-one setup where the router namespaces were created on the same system as the overlay endpoints?

In my case, the Overlay VMs are on one compute node and the L3 agents and router namespaces are running on another physical machine.

I'll try to collect the bridge and ip info you have asked for and post it later this evening.

Regarding the IP traffic, I did not see it being sent anywhere, but have not tracked down where exactly it got dropped. Will do that next.

Thanks,
-Uday

[Uday (nagaraj)]

Hi Carl,

Here's the info you asked for I've copied the relevant parts first, and the whole output later. There's a ton of output since my system is presently running about 20-25 VMs, so please bear with me.

The problem address here is 200.200.200.1. That has been programmed on eno1 (which is not even linked up)

+++++++++++++++++++++++

arp output from vm that is experiencing ping issues:

[root@ramu-2-vm ~]# arp -an
? (200.200.200.1) at 06:ab:a1:96:e5:09 [ether] on eth2
? (200.200.200.2) at fa:16:3e:9a:0e:bb [ether] on eth2
[root@ramu-2-vm ~]#

Info retrieved from Compute node:
brctl show:

qbr2b6f9879-b7 8000.06aba196e509 no qvb2b6f9879-b7
                                                        tap2b6f9879-b7

ip addr show:

2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 34:40:b5:db:26:e8 brd ff:ff:ff:ff:ff:ff
    inet 200.200.200.1/32 scope global eno1
       valid_lft forever preferred_lft forever

98: qvb2b6f9879-b7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr2b6f9879-b7 state UP qlen 1000
    link/ether 06:ab:a1:96:e5:09 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4ab:a1ff:fe96:e509/64 scope link
       valid_lft forever preferred_lft forever

ip route:

[root@uday-compute1-node ~]# ip route
default via 9.70.29.254 dev eno2 proto static metric 100
1.1.1.0/24 dev enp21s0f0 proto kernel scope link src 1.1.1.3
6.6.6.0/24 dev enp0s26f0u2 proto kernel scope link src 6.6.6.1
9.70.29.0/24 dev eno2 proto kernel scope link src 9.70.29.105
9.70.29.0/24 dev eno2 proto kernel scope link src 9.70.29.105 metric 100
[root@uday-compute1-node ~]#

+++++++++++++++++++++++++++++++++++++++

brctl show:
[root@uday-compute1-node ~]# brctl show
bridge name bridge id STP enabled interfaces
qbr1938e2c9-cf 8000.3ef18f60279f no qvb1938e2c9-cf
                                                        tap1938e2c9-cf
qbr2850e322-f0 8000.02ad7cc9c93c no qvb2850e322-f0
                                                        tap2850e322-f0
qbr2b6f9879-b7 8000.06aba196e509 no qvb2b6f9879-b7
                                                        tap2b6f9879-b7
qbr2c80eff4-d6 8000.e6ef21a4ed24 no qvb2c80eff4-d6
                                                        tap2c80eff4-d6
qbr30142252-b2 8000.86290fcffe3d no qvb30142252-b2
                                                        tap30142252-b2
qbr47aa100c-c9 8000.ae19fb36ede5 no qvb47aa100c-c9
                                                        tap47aa100c-c9
qbr52a374ed-83 8000.02e1ee9b62cb no qvb52a374ed-83
                                                        tap52a374ed-83
qbr5b4878c7-3f 8000.32942ffd7513 no qvb5b4878c7-3f
                                                        tap5b4878c7-3f
qbra3920a80-41 8000.2a47cc4f84f5 no qvba3920a80-41
                                                        tapa3920a...

[Uday (nagaraj)]

Hi,

Any comments or observations regarding the data I've posted?

Thanks,
-Uday

[Carl Baldwin (carl-baldwin)]

Uday, sorry I haven't had a chance to look at it today. Will try to get to it soon.

[Uday (nagaraj)]

Thanks Carl, will look forward to your comments.

Regards,
-Uday

[Uday (nagaraj)]

Hi,

Any comments on this? It has gone eerily quet!

Thanks,
-Uday

[Carl Baldwin (carl-baldwin)]

My initial attempt to reproduce was on an all-in-one system. I'd need to get on another system with an extra unused interface to try to reproduce. I'll see if I can get to one. It is hard right now because I'm at the QA mid-cycle.

I looked through your data from comment #14. I do not see any reason for arp to respond with the mac address of the bridge when 200.200.200.1 is queried. eno1 is not connected physically or virtually to any L2 or L3 parts of the rest of the network stack. The IP 200.200.200.1 doesn't seem to show up in any other context either. I'm a bit baffled.

This appears to be a bug in the linux networking stack. Can we boil it down to a simpler configuration, maybe not using Neutron or Openstack, to expose this behavior?

I'm still not convinced that this needs to be a security embargoed bug. So far, it just seems to be an incorrect arp response. I don't see how it exposes the underlying host to the tenant network. Does anyone else?

[Uday (nagaraj)]

Hi Carl,

I dont think this is a bug with the linux stack, as far as it is concerned, there is a valid IP address on the system (pinging that address from the host itself shows ping responses).

The issue here is really with the use of Linux bridges to achieve security groups. The linux bridge has a tap device which will allow ARP packets to be seen by the linux network stack. And this is the issue that I have been trying to highlight.

I tried this experiment without openstack and neutron (commands copied below) and see similar behavior.

If I could make a suggestion, is it possible to have iptables rules that would force packets coming in to the linux bridge to be sent only to the ovs patch port and not anywhere else? Would it help to NAT packets that match host interface addresses (and have ovs rules to modify addresses back before further processing?)

I am fine removing the security tag for this bug. But I do think this is a real flaw that should be addressed. Please let me know if there's something I can do to remove the security status. The flaw really is that overlay client will experience loss of connectivity if IP address clash between overlay and underlay interfaces.

Thanks,
-Uday

NOTE: IP address of eno1 is 10.0.0.5/24 for this experiment

  297 ip link add dev vm1 type veth peer name vm2
  298 ip link set dev vm1 up
  299 ip link add brm type bridge
  300 ip tuntap add tapm mode tap
  301 ip link set dev tapm up
  302 ip link set tapm master brm
  303 ip link set vm1 master brm
  304 ip addr add 10.0.0.1/24 dev brm
  305 ip link set brm up
  306 ip netns add nnsm
  307 ip link set vm2 netns nnsm
  308 ip netns exec nnsm ip addr add 10.0.0.2/24 dev vm2
  309 ip link set dev vm2 up
  310 ip netns exec nnsm ip link set dev vm2 up
  311 ip netns exec nnsm ip a

ip addr add dev eno1 10.0.0.5/24

[root@uday-compute1-node ~]# ip netns exec nnsm ping 10.0.0.5
PING 10.0.0.5 (10.0.0.5) 56(84) bytes of data.
64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 10.0.0.5: icmp_seq=2 ttl=64 time=0.060 ms
^C
--- 10.0.0.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.060/0.060/0.061/0.007 ms

[root@uday-compute1-node ~]# ip netns exec nnsm arp -an
? (10.0.0.5) at 16:55:36:1c:f0:f5 [ether] on vm2
? (10.0.0.1) at 16:55:36:1c:f0:f5 [ether] on vm2
[root@uday-compute1-node ~]#

[Carl Baldwin (carl-baldwin)]

It really shouldn't mean anything for the "system" to have an IP address. In Linux, IP addresses are associated with interfaces, not the system as a whole.

The only interface with 200.200.200.1 associated is eno1. When an arp request is seen on another interface, the system shouldn't respond for addresses that only appear on eno1 unless proxy_arp is enabled. Can you tell me if proxy_arp is enabled on this system? The follow command should show it:

$ sudo find /proc/ -name proxy_arp -exec echo {} \; -exec cat {} \;

I didn't think proxy_arp could be involved here at first because I didn't see an entry in the routing table for 200.200.200.1. However, I went to read the docs again [1] and found that "The device performing proxy ARP responds for all ARP queries on behalf of IPs reachable on interfaces other than the interface on which the query arrives." This could be the reason why you're seeing this. Proxy arp *should not* be enabled anywhere in an Openstack host as far as I'm concerned.

Please let me know what you find wrt proxy_arp.

[1] http://linux-ip.net/html/ether-arp-proxy.html

[Uday (nagaraj)]

Hi Carl,

Proxy arp is not enabled (please see output below).

eno1 and all other NICs are enabled, hence a ping that is intiated on the host itself will get responses, since the interface is up. It is just not linked up physically.

I have also provided the output for ifconfig -a too.

Thanks,
-Uday

[root@uday-compute1-node ~]# sudo find /proc/ -name proxy_arp -exec echo {} \; -exec cat {} \;
/proc/sys/net/ipv4/conf/all/proxy_arp
0
/proc/sys/net/ipv4/conf/br-int/proxy_arp
0
/proc/sys/net/ipv4/conf/br-tun/proxy_arp
0
/proc/sys/net/ipv4/conf/brm/proxy_arp
0
/proc/sys/net/ipv4/conf/default/proxy_arp
0
/proc/sys/net/ipv4/conf/eno1/proxy_arp
0
/proc/sys/net/ipv4/conf/eno2/proxy_arp
0
/proc/sys/net/ipv4/conf/enp0s26f0u2/proxy_arp
0
/proc/sys/net/ipv4/conf/enp21s0f0/proxy_arp
0
/proc/sys/net/ipv4/conf/enp21s0f1/proxy_arp
0
/proc/sys/net/ipv4/conf/enp26s0f0/proxy_arp
0
/proc/sys/net/ipv4/conf/enp26s0f1/proxy_arp
0
/proc/sys/net/ipv4/conf/enp36s0f0/proxy_arp
0
/proc/sys/net/ipv4/conf/enp36s0f1/proxy_arp
0
/proc/sys/net/ipv4/conf/lo/proxy_arp
0
/proc/sys/net/ipv4/conf/ovs-system/proxy_arp
0
/proc/sys/net/ipv4/conf/tapm/proxy_arp
0
/proc/sys/net/ipv4/conf/vm1/proxy_arp
0
[root@uday-compute1-node ~]#

----------------------------

[root@uday-compute1-node ~]# ifconfig -a
br-int: flags=4098<BROADCAST,MULTICAST> mtu 1500
        ether 12:d9:cf:d4:1b:48 txqueuelen 0 (Ethernet)
        RX packets 1184 bytes 88203 (86.1 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

br-tun: flags=4098<BROADCAST,MULTICAST> mtu 1500
        ether b6:e0:86:6a:de:48 txqueuelen 0 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

brm: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.0.0.1 netmask 255.255.255.0 broadcast 0.0.0.0
        inet6 fe80::1455:36ff:fe1c:f0f5 prefixlen 64 scopeid 0x20<link>
        ether 16:55:36:1c:f0:f5 txqueuelen 0 (Ethernet)
        RX packets 18 bytes 1040 (1.0 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 26 bytes 2630 (2.5 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 200.200.200.1 netmask 255.255.255.255 broadcast 0.0.0.0
        ether 34:40:b5:db:26:e8 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 370660 bytes 26453130 (25.2 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 9.70.29.105 netmask 255.255.255.0 broadcast 9.70.29.255
        inet6 fe80::3640:b5ff:fedb:26ea prefixlen 64 scopeid 0x20<link>
        ether 34:40:b5:db:26:ea txqueuelen 1000 (Ethernet)
        RX packets 5772073 bytes 373584706 (356.2 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 9934 bytes 1361389 (1.2 MiB)
        TX errors 0 d...

[Carl Baldwin (carl-baldwin)]

With proxy_arp disabled, this shouldn't be the behavior. I'm sorry that I haven't had time to try to reproduce this behavior. But, I'm back to thinking this is a kernel bug.

Let me see what I can do to find an appropriate test system to try to reproduce this issue. Many of ours are in use for other testing but I think I can get access to something.

[Uday (nagaraj)]

Thanks for your response.

Regards,
-Uday

[Uday (nagaraj)]

Hi Carl,

Were you able to find anything wrong with the configs I sent you?

Thanks,
-Uday

[Aaron Rosen (arosen)]

Hi Uday,

I'm going to try and reproduce this too. It seems like you're saying that the linuxbridge interface that is connected to ovs is actually causing patches to route via the hosts network stack? Let me see if i can reproduce this.

[Uday (nagaraj)]

Hi Aaron,

Thanks for looking into this. Yes, due to the linux bridge packets are being processed by the host IP stack.

One thing in our setups here, the target of the overlay VM's ARP packet is located on a remote host, and I think that makes a big difference. The target IP also happens to be an IP address on the local host.

Thanks,
-Uday

[Uday (nagaraj)]

Hi Aaron,

One additional point, I'm not sure if the host stack is actually routing. What is see is that the host stack is responding to ARPs. The reason may be that broadcast packets received on the tap interfaces also get to the IP stack

I didnt see unicast packets having the same issue. No idea if multicast will see some odd behavior too.

Thanks,
-Uday

[Jeremy Stanley (fungi)]

The behavior you're describing is configurable through setting sysctl knobs like net.ipv4.conf.all.arp_ignore=1 and net.ipv4.conf.all.arp_announce=2, but by default yes the Linux kernel does reply to ARP queries on any interface for addresses it has configured on any other interface. This is probably system hardening we should cover in documentation, at least in the Security Guide, if we don't do so already.

[Carl Baldwin (carl-baldwin)]

I agree arp_ignore=1 makes sense [1]. I thought proxy_arp would apply here but, reading it again, proxy_arp is for proxying subnets reachable through another interface. In this case, it is a local address configured on another interface. Try setting it to 1.

I'm not sure what the arp_announce setting will change in this case.

[1] http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP

[Aaron Rosen (arosen)]

I wasn't able to resolv the mac of the underlying host from the guest (and my settings here are set to 0 the default). Getting a multi node setting working to confirm if this maybe occurs in the other direction for some reason.

[Aaron Rosen (arosen)]

I'm not able to reproduce this in the multinode setup either. Are you sure that the traffic isn't just routing through the l3-agent and hitting the ip on the other side of the gateway (which would be the expected behavior).

[Uday (nagaraj)]

Hi,

@Jeremy: Your idea helped, I set arp announce and ignore on the physical NIC as well as the bridge where the VM connects and this works fine. Thanks for the suggestion

@Aaron: The traffic was definitely not going out of the host, earlier on in this thread I have detailed out the MAC addresses and ARP resolutions that show the error clearly. I am using RHEL 7.1 (Linux is 3.10.0-229.el7.x86_64). Could that have a bearing?

I'm good with the documentation part. Please let me know if there's any further info needed.

Thanks,
-Uday

[Carl Baldwin (carl-baldwin)]

I think we have a pretty good handle on this. I also am of the opinion that this bug does not need to be security embargoed. What do others think? Can we make this public?

[Jeremy Stanley (fungi)]

If other Neutron core security reviewers are comfortable with switching this report to public security (or more likely just public with a security tag to denote a hardening or documentation opportunity) then I certainly don't object.

[Uday (nagaraj)]

Hi,

Do I need to make the security setting changes or will someone from the security team make the change?

Thanks,
-Uday

[Aaron Rosen (arosen)]

@Uday, perhaps it is related to your kernel version. I wasn't able to reproduce it with

3.16.0-45-generic #60~14.04.1-Ubuntu SMP

I think we can make this public as well. Thanks for the report though Uday definitely an interesting find.

[Jeremy Stanley (fungi)]

I've marked the security advisory task "won't fix" but added the security tag in case this corner case may be of interest to OSSN and Security Guide editors.

[Armando Migliaccio (armando-migliaccio)]

This bug is > 240 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]