nova security group iptables rules conflict for overlapping subnets

[Jiajun Liu]

in my devstack all in one installation, I created two tenant, and each tenant has a subnet (10.0.0.0/24), and then start two instance each tenant, after instance started up. I check the iptable rules.

instances of tenant1:
# nova list
+--------------------------------------+-------+--------+---------------+
| ID | Name | Status | Networks |
+--------------------------------------+-------+--------+---------------+
| a0c528b9-c5cc-49b9-aa5c-f680b739f385 | test1 | ACTIVE | net1=10.0.0.3 |
| 7f144411-dcee-4672-bb5d-e4b1d19677dd | test2 | ACTIVE | net1=10.0.0.4 |
+--------------------------------------+-------+--------+---------------+

instances of tenant2:
+--------------------------------------+-------+--------+---------------+
| ID | Name | Status | Networks |
+--------------------------------------+-------+--------+---------------+
| 880a0ce4-eae4-4850-9684-32b5de8a6eca | test1 | ACTIVE | net1=10.0.0.3 |
| 60b40131-2070-4090-9720-e77579e43de7 | test2 | ACTIVE | net1=10.0.0.4 |
+--------------------------------------+-------+--------+---------------+

iptable rules:

# iptables-save
...
-A nova-compute-local -d 10.0.0.3/32 -j nova-compute-inst-1
-A nova-compute-local -d 10.0.0.4/32 -j nova-compute-inst-2
-A nova-compute-local -d 10.0.0.3/32 -j nova-compute-inst-3
-A nova-compute-local -d 10.0.0.4/32 -j nova-compute-inst-4
...

from the iptable rules, we can see that the instances with the same ip address will go through the same iptable rules which is not as we expected. More important, one tenant's security group setting may impact other tenant's.

[dan wendlandt (danwent)]

Yes, all parts of nova assumes that IPs can never overlap. This is similar to the issues with nova-metadata server and overlapping IPs.

During Folsom we had wanted to get security groups ported over to Quantum, which would have meant that we add support for overlapping IPs as well as multiple interface per VM. However, we ran out of time, and so we have the partial solution of utilizing Nova security groups, with those limits. Not great, I agree.

To me, the real concern here is that there is no good way to make sure nova metadata & security groups are only used if Quantum has no overlapping IPs. One thing we had discussed was potentially having a flag in Quantum that indicates whether overlapping IPs should be allowed or not. We could default this to false, and then mention the limitations around metadata + security groups where we document how to enable overlapping IPs in Quantum. The concern I have with this approach is that defaulting to not allowing overlapping IPs seems like the wrong long-term default for quantum, once quantum itself implements security groups (and we modify the metadata mechanism to handle overlapping IPs). Outside of that, I think this is just a question of documentation, unless someone else sees anything super bright.

[dan wendlandt (danwent)]

So there are two high-level approaches to this.

1) try to patch nova security groups to handle overlapping IPs.
2) just focus on finishing the existing design of security groups within quantum, which implicitly will handle overlapping IPs.

Some quick thoughts based on how we might perform #1 in a non-invasive way:
- add an ensure namespace method to linux-net
- add a delete namespace method to linux-net
- for LibvirtHybridOVSBridgeDriverin nova/virt/libvirt/vif.py, ,plug() create the bridge and bridge-side veth device and place them into a instance specific (or vif-specific) namespace. We could do something similar for the BridgeDriver to support Linux Bridge plugin.
- modify nova/virt/firewall.py, and create a NamespaceIPtablesFirewallDriver. This driver would be a "meta driver", creating a separate instance of the normal IPtablesFirewallDriver for each namespace.
- when an instance id destroyed, destroy the instance of IPtablesFirewallDriver for that instance/vif.
- when a vif is unplugged(), destroy the corresponding namespace.

[dan wendlandt (danwent)]

Note: as a safety measure on the quantum side, I would consider adding a flag to prevent inadvertently creating subnets that overlap.

[dan wendlandt (danwent)]

for folsom, we added a flag and new default setting for quantum to avoid overlapping IPs unless explicitly enabled.

in grizzly quantum will implement its own security groups, so moving this to won't fix.

[dan wendlandt (danwent)]

seems like I do not have the permissions to set won't fix for nova. hoping someone else will...