In multirack environment its impossible to create 3 nated networks with connectivity to each other

[Sergey Yudin]

This set of filter rules being created by 2 nated networks.

-A FORWARD -d 10.0.13.0/24 -o fuelbr18200 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.0.13.0/24 -i fuelbr18200 -j ACCEPT
-A FORWARD -i fuelbr18200 -o fuelbr18200 -j ACCEPT
-A FORWARD -o fuelbr18200 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i fuelbr18200 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -d 10.0.14.0/24 -o fuelbr18201 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.0.14.0/24 -i fuelbr18201 -j ACCEPT
-A FORWARD -i fuelbr18201 -o fuelbr18201 -j ACCEPT
-A FORWARD -o fuelbr18201 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i fuelbr18201 -j REJECT --reject-with icmp-port-unreachable

In this configuration connectivity from .14.0/24 to 13.0/24 will never work because traffic will be blocked by
-A FORWARD -o fuelbr18200 -j REJECT --reject-with icmp-port-unreachable

e.g. packet from 10.0.14.3 inport fuelbr18201 to 10.0.13.3 outport fuelbr18200 will be rejected.

[Andrey Nikitin (heos)]

Looks like this bug is related to fuel-devops developed by Fuel QA Team.

[Artem Panchenko (apanchenko-8)]

It's known limitation, traffic to NATed networks can't be routed, see libvirt docs [0]:

[forward]
[nat]
...
"Inbound connections from other networks are all prohibited; all connections between guests on the same network, and to/from the host to the guests, are unrestricted and not NATed"

If you want to create networks for multirack environment, please use 'route' forward mode. In case you need to provide Internet access via such networks, you should manually add custom iptables rules for SNAT.

[0] https://libvirt.org/formatnetwork.html#elementsMetadata

[Sergey Yudin (tsipa740)]

If thats known limitation of libvirt than don't use libvirt or use custom rules.

This prevents us from testing multirack environments with controllers located in different racks.