adding 0.0.0.0/0 address pair to a port bypasses all other vm security groups

[Alex]

On an openstack-ansible / newton setup with linuxbridge, a customer ran:

neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0

to bypass the ip source restriction (pfsense router and had to route packets).

The impact of running the above, was an allow all rule was added to all ports in the network, bypassing all security groups.

The iptables rule:

 905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src

used on all ports, now triggers as:

0.0.0.0/1
128.0.0.0/1

was added to the ipset NIPv44046d62c-59c8-4fd0-a547 (verified by looking at the ipset on the nova hosts). Removing the two lines from the ipset restored all security groups.

Expected result was to remove ip filtering on the single port.

This sounds similar to:

https://bugs.launchpad.net/neutron/+bug/1461054

but is marked fixed long ago.

I've marked this as a security bug as a change to a single port can bypass other ports security groups.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Jeremy Stanley (fungi)]

Alex, are you saying an account for one tenant/project inserting this rule can bypass filtering for the instances/ports belonging to others? Or only for their own? If the latter, I think the sooner we switch this to a public workflow the better so that users will know to avoid doing this even before it gets patched.

[Alex (akrohn)]

Jeremey, yes, the latter. The only security risk here is that a user who can modify a port on a single instance can bypass security groups on all other instances within the same network (so does not cross tenants / networks). Not sure if that qualifies or not, but thought it better safe then sorry, so please feel free to open publicly if you are comfortable with it.

[Slawek Kaplonski (slaweq)]

@Alex:

Are You sure it added rule:

905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src

to all ports in same network or to all ports which uses same security group?

[Alex (akrohn)]

Slawek, I'm not 100% sure as those are the one and the same in this environment. You might be right and it impacted all instances using the default security group.

However, the add allowed-address-pairs didn't add the iptables rule, it already existed. This rule:

905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src

I believe comes from the default security group, and the ipset contains a list of all instance ip's on the network and is what allows instances to talk to each other by default (so traffic sourced from any instance on the same network is allowed).

The add allowed-address-pairs added 0.0.0.0/1 and 128.0.0.0/1 to the ipset, thereby putting a default allow from anywhere in place, and bypassing any security groups.

At least, I'm pretty sure that's what happened. We do have a staging environment and can test anything out. =)

[Brian Haley (brian-haley)]

I just wanted to clarify a few things.

1. You are using the iptables hybrid firewall driver, i.e. in ml2_conf.ini:

firewall_driver = iptables_hybrid

Not sure if the same issue exists using 'openvswitch' as the firewall_driver.

2. We should identify the correct way to get what you want, for example:

$ openstack port set --disable-port-security <port_id>

3. I think in this case the ipset wound up being shared amongst the ports from this tenant on the compute node, right? Other compute nodes with ports in this subnet are probably not affected? I haven't tried to reproduce this locally yet so just assuming.

4. Possible workaround is to disable the use of ipset by changing this in ml2_conf.ini:

enable_ipset = false

performance will be impacted though.

[Slawek Kaplonski (slaweq)]

I didn't try to reproduce this yet but I think that what happened here is:
1. By default in security group there is rule to accept ingress traffic from all other ports which uses same security group,
2. When You added this allowed_address_pair, security group rules (allowed source IPs) where updated for all other ports which use same SG - and because of that You had such bad rule in iptables.

If I am right, You should be able to remove this wrong rule from all ports by just removing from security group rules which allows traffic from "remote_group_id".

Also if that is the problem, disabling ipsets will not help to workaround this as such rules still will be added directly in iptables chains.

And also I think that in such case openvswitch firewall driver will also be impacted.

[Alex (akrohn)]

From #6:

1. We are using iptables driver:

/etc/neutron/plugins/ml2/linuxbridge_agent.ini:firewall_driver = iptables

2. The end goal is to remove the source ip filtering; allowing the ports for the one router instance to be able to send packets source from different ip's (think pfsense doing an ipsec tunnel to remote network). I think yes, just disabling port security might be a workaround.

3. No, it impacted instances across all compute nodes. The 0.0.0.0/1 and 128.0.0.0/1 existed in the NIPv44046d62c-59c8-4fd0-a547 ipset's across all nodes

From #7

That sounds right. My take on it is I think what happened is:

1. We have a default security group:

ALLOW IPv6 from default
ALLOW IPv4 from default
ALLOW IPv4 to 0.0.0.0/0
ALLOW IPv6 to ::/0

on all instances.

2. The firewall driver adds this iptables rule for the default security group:

905K 55M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv44046d62c-59c8-4fd0-a547- src

and also adds a matching ipset that contains all other ip's on the network. This allows inbound traffic sourced from other instances on the same network to reach a particular instance.

3. Running:

neutron port-update $port-uuid --allowed-address-pairs type=dict list=true ip_address=0.0.0.0/0

added 0.0.0.0/1 and 128.0.0.0/1 to the ipset NIPv44046d62c-59c8-4fd0-a547- on all compute nodes which caused the unexpected behavior.

[Akihiro Motoki (amotoki)]

I confirmed what is reported here with linuxbridge mechanism driver.
http://paste.openstack.org/show/730395/

After setting an allowed-address-pair (to vm1),
* 0.0.0.0/0 is added as a valid IP address range (L.165)
* 0.0.0.0/1 and 128.0.0.0/1 are added to the ipset NIPv41de12a7a-aae7-4bac-8fb3- (L.202-214)

[Akihiro Motoki (amotoki)]

[analysis of what happens]

The issue which is reported as a security bug is about the latter ipset.
This is broken down into two cases:
- (a) ingress security group rule for vm1
- (b) ingress security group rule for VMs other than vm1

For (a), this is questionable.
The concept of the allowed-address-pair is to allow IP addresses from a specified range for a port (which affects "egress" check).
However it is not expected to affect "ingress" check. (<-- problem 1)

For (b), this is an expected behavior. vm1 port can use any IP address and "default" security group declares to accept all ingress traffic from the "default" security group members.
Even though this is an expected behavior, it is still a bit surprising to users. I believe most users do not imagine security groups of vm2-vm4 (servers other than vm1) will be full open. (<-- problem 2)

[Akihiro Motoki (amotoki)]

For problem 2 in #10, I don't have a good solution directly.
What I can suggest is not to use "remote group" in security group rules and "allowed-address-pair" (with a wider IP address range) together. It is recommended to use "IP address" rather than "remote group" in security group rules if you use "allowed-address-pair".

For problem 1 in #10, we can potentially use a separate ipset for each port. Each ipset contains a list of IP addresses of security group members but it does not contain its own IP address of the port.

However, if we go to the above route for problem 2, problem 1 will be minor because problem 1 also occurs only when using "remote group" and "allowed-address-pairs" together.

My current conclusion is to add warnings on this to various documents like neutron API reference, security notes and more if any.

Thought?

[Slawek Kaplonski (slaweq)]

I agree with Akihiro's proposal here. IMO best thing we can do here is add warnings to documentation to make operators aware of this kind of risk.

[Jeremy Stanley (fungi)]

Thanks everyone for the prompt analysis. I've triaged this as a class B2 port per the OpenStack VMT taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Alex (akrohn)]

Thanks Akihiro, your description makes perfect sense and I see now why it did what it did. Definitely worth a note in the doc.

[Boden R (boden)]

Based on previous comments it appears we plan to doc this in the neutron api-ref.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/661594

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661594
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=e02c4b214b6762823c5e1ae5719f08f5f51910e8
Submitter: Zuul
Branch: master

commit e02c4b214b6762823c5e1ae5719f08f5f51910e8
Author: Slawek Kaplonski <email address hidden>
Date: Mon May 27 02:45:15 2019 +0200

    [api-ref] Add short warning about ANY IP address in allowed address pair

    Change-Id: Ie3ed5ea81e0df50a699174fef95eb32337ed5682
    Closes-Bug: #1793029

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-lib 1.28.0 release.