[RFE] Grouping remote_ip_prefix by ipset

[Takao Indoh]

This RFE is requesting a feature for grouping IP addresses of remote_ip_prefix using ipset to improve network performance.

[Background]
To allow access from/to specific IP address, remote_ip_prefix can be used in security group rule. This is used for:
o To allow access from/to machines which is outside OpenStack environment.
o To allow access between datacenter in multi-site environment.

[Existing problem]
For the usage above, We have to specify IP address(CIDR) as remote_ip_prefix one by one and the number of security group rules are increased. It caused performance degradation of network.

We had the same problem about remote_group_id and it is solved by enable_ipset feature[1]. This feature improves network performance by grouping IP addresses which belong to the same security group. I want to extend this feature for remote_ip_prefix.

[Proposal]
To solve the problem above, I propose the following feature.

1) Introduce new feature to group many CIDRs into one set and specify the set as remote_ip_prefix collectively.
2) Improve neutron and L2 agent so that L2 agent updates iptables using ipset for this set of CIDRs.

By this feature, we can reduce the number of security group rules when many remote_ip_prefix is used.

When we specify IP address to allow access, we need to do like this:

neutron security-group-rule-create --remote-ip-prefix 192.168.100.10/32
neutron security-group-rule-create --remote-ip-prefix 192.168.101.15/32
neutron security-group-rule-create --remote-ip-prefix 192.168.102.20/32
...

Therefore many security group rules are generated and it causes degradation of network performance. I propose new feature to group these CIDRs into one set and specify this set as remote_ip_prefix.

And then, L2 agent converts this CIDR set to the ipset group and apples it to iptables. This is the same manner as converting remote_group_id to ipset group.

[1]
https://specs.openstack.org/openstack/neutron-specs/specs/juno/add-ipset-to-security.html

[Armando Migliaccio (armando-migliaccio)]

I must be missing something but the IP consolidation may happen even without making API changes. Are you sure that this is a degradation problem worth fixing? Do you have hard data that shows evidence of performance degradation at scale?

[Assaf Muller (amuller)]

@Miguel, can you please help triage this RFE?

[Yushiro FURUKAWA (y-furukawa-2)]

Hi armando,

I'm co-worker of Takao Indoh. Here is a benchmark for ipset. https://goo.gl/photos/sLpb7dgpm6ZPvM9SA
Please refer it.

[Carl Baldwin (carl-baldwin)]

From discussion after the drivers' meeting in openstack-meeting, it seems this benchmark is just showing the benefits of ipset. Is it really relevant to this request?

[Mickey Spiegel (emspiege)]

Please clarify whether the proposal is:
1) Change the security groups API to allow more than one remote_ip_prefix in a rule, or
2) Add an address_group as an alternative to remote_ip_prefix, or
3) Coalesce multiple rules in the same security group, and program them as one iptables rule (no API changes)

[Miguel Angel Ajo (mangelajo)]

I think the most reasonable option could be 3, where we add as many rules as necessary (without modifying the model), and then the iptables firewall is smart enough to coalesce all of the remote addresses in ipsets.

@armax, @amuller, @carl-baldwin: currently, as far as I remember, the ipset implementation coalesces in groups sets of ips when a rule is pointing to a remote_group_id, but not in other cases.

This is not very simple to address anyway, because we need to find the rules that have common matchings and still reference the same sets of addresses, and then put those addresses in named ipsets, and reference the ipset.

For example, if we have:

neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.100.10/32
neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.101.15/32
neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.102.20/32

It's very easy, but what if we have:

neutron security-group-rule-create sg1 --protocol icmp --remote-ip-prefix 192.168.100.10/32
neutron security-group-rule-create sg1 --protocol icmp --remote-ip-prefix 192.168.101.15/32
neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.100.10/32
neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.101.15/32
neutron security-group-rule-create sg1 --remote-ip-prefix 192.168.102.20/32

We can only optimize that into two rules by identifying the two sets of addresses, one would be

ICMP + ipset1
the other, just ipset2

where ipset1 = {192.168.100.10/32, 192.168.101.15/32}
and ipset2 = {192.168.100.10/32, , 192.168.101.15/32, 192.168.102.20/32}

[Takao Indoh (tindoh)]

@carl-baldwin:
Yes, the benchmark in #3 is a result of ipset performance test. This RFE is improving performance using ipset, hence I believe we can get same improvement as this result once this feature is implemented.

@emspiege, @mangelajo:
Regarding the implementation, at first I thought proposal like option 2) in #5, but now I'd like to propose improving by option 3). What I'm thinking is L2 agent integrates rules using ipset if they are the "same". The "same" means they have the same direction, protocol, port_range_min/max, and ethertype.

[Kevin Benton (kevinbenton)]

Without an API change this is essentially just and optimization local to the agent code so it doesn't really need an RFE and can be handled like a normal performance bug IMO.

[Miguel Angel Ajo (mangelajo)]

Not sure if RFE or bug, but it's clear to me option #3 is the most reasonable,
It's a performance optimization on the L2 agent.

[Armando Migliaccio (armando-migliaccio)]

Yes. If someone can sheperd this bug fix through review, the let's go for it!

[Armando Migliaccio (armando-migliaccio)]

Let's evaluate the complexity of the code to justify if the performance gain is worth the maintenance pain.

[Takao Indoh (tindoh)]

Ok, I'm investigating source code to figure out a simple way to implement this. I'll post it once I finish.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.