Rocky DVR-SNAT seems missing entries for conntrack marking

[Ewald van Geffen]

Hello,

I've been playing with Rocky on CentOS 7 with DVR / DVR-SNAT / BGP. Our provider network uses private IP-space for scalability. Our tenants run in public IP-space.

Steps:

openstack network create --share --provider-network-type vlan --external --provider-physical-network EXTERNAL NET-EXT-VLAN625-UPLINK --provider-segment 625
openstack subnet create --network NET-EXT-VLAN625-UPLINK --subnet-range 192.0.2.0/24 SUBNET-EXT-VLAN625-UPLINK
openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-EXT-VLAN625-UPLINK

openstack address scope create --share --ip-version 4 SCOPE-SHARED-VFOUR
openstack subnet pool create --pool-prefix 93.115.169.128/25 --address-scope SCOPE-SHARED-VFOUR POOL-SHARED-EXTERNAL
openstack subnet pool set --default-prefix-length 28 --min-prefix-length 26 POOL-SHARED-EXTERNAL

openstack network create --share NET-INT-PUBLIC-SHARED
openstack subnet create --network NET-INT-PUBLIC-SHARED --subnet-pool POOL-SHARED-EXTERNAL --prefix-length 28 SUBNET-INT-PUBLIC-SHARED
openstack bgp speaker add network BGP-REFLECTOR-SHARED NET-INT-PUBLIC-SHARED

openstack router create ROUTER-SHARED
openstack router set ROUTER-SHARED --disable-snat --external-gateway NET-EXT-VLAN625-UPLINK
openstack router add subnet ROUTER-SHARED SUBNET-INT-PUBLIC-SHARED
openstack bgp speaker list advertised routes BGP-REFLECTOR-SHARED

Result:

+-------------------+-----------+
| Destination | Nexthop |
+-------------------+-----------+
| 93.115.169.128/28 | 192.0.2.6 |
+-------------------+-----------+

Where 192.0.2.6 is the IP in the SNAT instance on the DVR-SNAT network node. So far things seem good however it seems my egress traffic is blocked. I'm not 100% sure it's not caused by misconfiguration but I found the two lines to get traffic flowing in my config:

https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1093
was rewritten to "dont_block_external = (ip_version == lib_constants.IP_VERSION_4 and external_port)"

-> This would allow traffic to go egress into the provider network

https://github.com/openstack/neutron/blob/stable/rocky/neutron/agent/l3/router_info.py#L1122 if-return statement was commented out.

-> I need that connmark/mark restore rule to have ingress replies match the existing connections.

Validate:

ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t mangle -L neutron-l3-agent-POSTROUTING -nv
Chain neutron-l3-agent-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
508 27174 CONNMARK all -- * qg-b075d908-66 0.0.0.0/0 0.0.0.0/0 connmark match 0x0/0xffff0000 CONNMARK save mask 0xffff0000

1 entry

ip netns exec $(ip netns | grep snat | awk '{ print $1}') iptables -t filter -L neutron-l3-agent-scope -nv
Chain neutron-l3-agent-scope (1 references)
pkts bytes target prot opt in out source destination
6588 350K DROP all -- * sg-c46c9df8-06 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4010000/0xffff0000

1 entry (instead of two before)

[Brian Haley (brian-haley)]

This could be a mis-configuration.

The address scopes of the internal and external subnets don't match, so I don't believe the proper iptables rules will be installed.

From the top part of https://docs.openstack.org/neutron/stein/admin/config-address-scopes.html (rocky has similar wording), it says:

3. Make sure that subnets on an external network are created from the subnet pools created above:

So they're using the same subnetpool to create a subnet for the external network as the internal ones, which should result in the same address scope on each.

If that doesn't help, please update the bug and we'll have to do further triage.

[Ryan Tidwell (ryan-tidwell)]

neutron-dynamic-routing is finding the route to the tenant network, so address scopes inside and outside are matching. What happens if you disable SNAT on that router?

[Akihiro Motoki (amotoki)]

According to the bug description, the external network belongs to an implicit address scope and the internal address scope belongs to SCOPE-SHARED-VFOUR. Traffic between these networks need to be NAT-ed but NAT is disabled for the router.

It looks like you need to enable NAT for the router or if you would like to forward traffic without NAT you need to make the external and internal networks belong to a same address scope.

Related to Ryan's comment, I am not sure what happens...

[Ewald van Geffen (cloudkitten)]

When I include 192.0.2.0/24 into the address-scope it starts working as intended. Great!

@Ryan: The initial post was about no SNAT + DVR + lack of pool configuration for external network. When I enable SNAT in this setup, it would've worked except 192.0.0/24 isn't globally routed so I never got further that the default gateway on the TOR 192.0.2.1/24, due to lack of return route towards TOR from Internet.

@Akihiro: I'm not entirely sure why the BGP announcement was made if my address-scope config was incomplete. Is there more documentation on "the external network belongs to an implicit address scope" ? Is this desirable behaviour?

[Brian Haley (brian-haley)]

Ewald - when you say "When I include 192.0.2.0/24 into the address-scope it starts working as intended", what did you change? Would just be good to have that in case it happens to others. Thanks.

[Ryan Tidwell (ryan-tidwell)]

I do see an issue here, and it is that you reported SUBNET-EXT-VLAN625-UPLINK was created without an address scope and running "openstack bgp speaker list advertised routes BGP-REFLECTOR-SHARED" showed you a route for 93.115.169.128/28. That should not happen, advertisements should only be made if the address scopes of both the external network and the tenant network match.

[Ewald van Geffen (cloudkitten)]

@Brian #5

openstack subnet pool create --pool-prefix 192.0.2.0/24 --address-scope SCOPE-SHARED-VFOUR POOL-SHARED-UPLINK
openstack subnet pool set --default-prefix-length 27 --min-prefix-length 25 POOL-SHARED-UPLINK
openstack subnet create --network NET-EXT-VLAN625-UPLINK --subnet-pool POOL-SHARED-UPLINK --prefix-length 27 SUBNET-EXT-VLAN625-UPLINK

TL;DR: internal _and_ external subnets need to be generated from subnet-pool(s) linked to the one and the same address-scope.

@Ryan #6,
That's exactly what I meant. I'd like to get this fixed/investigated so others don't run in BGP + DVR + NAT because that _would_ work, just not in my situation where external was publicly unroutable.

I think (based on zilch) it's some default code path behaviour for when you're dealing with multiple address scopes and a opportunistic decision had to be made to not break legacy configs.

[Ryan Tidwell (ryan-tidwell)]

I've created a few test cases to mimic your setup, and I can't reproduce the issue. A large number of tests assert the behavior when you have 2 different address scopes in play. What's missing is coverage on the case where you have the null address scope on either the external or tenant network and an address scope on the the other. I have written a few automated tests that cover this case, and they pass without any tweaks to the route lookups in neutron-dynamic-routing. I also am unable to reproduce this manually. Am I missing something? Both automated and manual tests are unable to reproduce this behavior.

[Ewald van Geffen (cloudkitten)]

Hi Ryan,

If you let me know what files/traces/captures you need for troubleshooting I'll provide them, I just don't know what you exactly need for further analysis.

I'm running on CentOS Linux release 7.6.1810 with openstack-neutron 13.0.3-1.el7 (bgp dragent on 13.0.3). I have three compute nodes w/ DVR, two network nodes with SNAT-DVR, BGP-dragent,...

Yours sincerely,
Ewald van Geffen

[Ewald van Geffen (cloudkitten)]

May close / it has little to no prio.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]