rules referencing security group members expose VMs in overlapping IP scenarios

[Kevin Benton]

create SG1 an SG2 that only allow traffic to members of their own group
create two networks with same 10.0.0.0/24 CIDR
create port1 in SG1 on net1 with IP 10.0.0.1
create port2 in SG1 on net2 with IP 10.0.0.2
create port3 in SG2 on net1 with IP 10.0.0.2

port1 can communicate with port3 because of the allow rule for port2's IP

This violates the constraints of the configured security groups.

Another incarnation of the bug happens if you:

(graphic representation: https://bugs.launchpad.net/neutron/+bug/1463589/+attachment/4416693/+files/sg-disjoint-networks-bug-with-router.png)
create SG1 and SG2, that only allow traffic to members of their own group
create two network (N1, N2) segments
create another network segment (N3)
add a router R that connects the N1 to N3

then add IPa, IPb to SG1 on N1
add IPc, IPd to SG1 on N2

then add IPc and IPd to SG2 on N3

IPa, and IPb will accept traffic from ports with IPc and IPd on SG2 even if they should not.

[Kevin Benton (kevinbenton)]

Relevant mail thread: http://lists.openstack.org/pipermail/openstack-dev/2015-June/066225.html

[Carl Baldwin (carl-baldwin)]

From the discussion in the thread that Kevin linked.

"Our security group grouping model implies that you are allowing specific neutron ports. That happens to be rendered into iptables rules based on the IP addresses, which violates the API intent whenever ports land in different overlapping subnets."

We attach security groups to an L2 thing but we expect it to apply across L3 routing boundaries. I think this is a fundamental flaw in the design of security groups. Or, at least, it is fundamentally at odds with the ability to create networks that overlap ip addresses. I think it is just a bad idea to try to consider an L2 object beyond the scope of the L2 network to which it is connected. A solution which involves tracing L3 connectivity has already been called "horrific" in the neutron team meeting [1]. +1k to that. I hope that we're not going to seriously consider it. It will be very painful to write, maintain, and scale.

[1] http://eavesdrop.openstack.org/meetings/networking/2015/networking.2015-06-08-21.01.log.html#l-62

[Carl Baldwin (carl-baldwin)]

I think we should examine the use cases around having networks with overlapping IP addresses and develop some guidelines for how to use it safely with multiple security groups. I just can't see how we can solve this problem in all cases.

[Kevin Benton (kevinbenton)]

>We attach security groups to an L2 thing but we expect it to apply across L3 routing boundaries. I think this is a fundamental flaw in the design of security groups.

I think it's the only feature of security groups that makes it somewhat user-friendly. It's an awesome concept to not have to keep track of the IP addresses of all of your instances. Our implementation just happens to not take overlapping IPs into account. That's not a problem with the API, it's a problem with how we rendered the groups into rules.

[yalei wang (yalei-wang)]

why the VMs should be isolated from each other if they are in different SGs, even if the rules is the same.

SG just control the pkg in/out the VM, if the different SGs have the same rules, I think VMs should have the same pkgs control, they could communicate each other if they are in the same network, like the port1 and port3 in this case.

[Kevin Benton (kevinbenton)]

They should be isolated because there is no rule that allows the traffic. The API rules are based on allowing a security group, so only members of that security group should be allowed to communicate with the VM.

[yalei wang (yalei-wang)]

got it, thanks Kevin for your detailed explanation.

[Kevin Benton (kevinbenton)]

I attached a visual representation of the issue.

[huaxiang (fanhuaxiang)]

Is this bug going to be solved?

[Armando Migliaccio (armando-migliaccio)]

This bug is > 240 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]