keepalived virtual_routes wrong order

[Florian Engelmann]

Neutron version: 13.0.6 (I will try to test this with version Zed as well)
ML2: OVS

1. create a provider network (eg. named: public)
2. create a subnet pool (eg. 100.100.100.32/27)
3. create a 1st subnet /29 from that subnetpool on network "public": 100.100.100.32/29 --gateway 100.100.100.33
4. create a 2nd subnet /29 from that subnetpool on network "public": 100.100.100.40/29 --gateway 100.100.100.33

NOTE: the "physical" gateway of the whole subnetpool is 100.100.100.33
-> so the GW of the 2nd subnet is in the range of the 1st subnet!

neutron_l3_agent will create a keepalived.conf like:

global_defs {
    notification_email_from <email address hidden>
    router_id neutron
}

vrrp_script ha_health_check_186 {
    script "/var/lib/neutron/ha_confs/f9ed7361-29b2-48e1-a96b-1a2919062021/ha_check_script_186.sh"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VR_186 {
    state BACKUP
    interface ha-f3350150-28
    virtual_router_id 186
    priority 50
    garp_master_delay 60
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass somepass
    }
    track_interface {
        ha-f3350150-28
    }
    virtual_ipaddress {
        169.254.0.186/24 dev ha-f3350150-28
    }
    virtual_ipaddress_excluded {
        192.168.199.1/24 dev qr-24c07a36-4f
        100.100.100.34/32 dev qg-7b9963a7-72
        100.100.100.42/32 dev qg-7b9963a7-72
        100.100.100.43/29 dev qg-7b9963a7-72
        fe80::xxxx:xxxx:xxxx:xxxx/64 dev qg-7b9963a7-72 scope link
        fe80::xxxx:xxxx:xxxx:xxxx/64 dev qr-24c07a36-4f scope link
    }
    virtual_routes {
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72
        100.100.100.32/29 dev qg-7b9963a7-72 scope link
    }
    track_script {
        ha_health_check_186
    }
}

So keepalived will try to create the default route BEFORE the route "100.100.100.32/29 dev qg-7b9963a7-72 scope link" has been created. This will throw an error like:

Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) Receive advertisement timeout
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) Entering MASTER STATE
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting VIPs.
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting E-VIPs.
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting Virtual Routes
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: Netlink: error: Network is unreachable(101), type=RTM_NEWROUTE(24), seq=1674751752, pid=0
Jan 26 17:49:56 xxxxxxxx Keepalived_vrrp[1532]: (VR_186) Sending/queueing gratuitous ARPs on ha-f3350150-28 for 169.254.0.186
...

And the default route will be missing:

169.254.0.0/24 dev ha-a1f79365-fc proto kernel scope link src 169.254.0.186
169.254.192.0/18 dev ha-a1f79365-fc proto kernel scope link src 169.254.192.14
192.168.199.0/24 dev qr-24c07a36-4f proto kernel scope link src 192.168.199.1
100.100.100.32/29 dev qg-7b9963a7-72 scope link
100.100.100.40/29 dev qg-7b9963a7-72 proto kernel scope link src 100.100.100.43

Changing the order of "virtual_routes" will fix this issue:

    }
    virtual_routes {
        100.100.100.32/29 dev qg-7b9963a7-72 scope link
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72
    }

The error message is now gone:

Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) Receive advertisement timeout
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) Entering MASTER STATE
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting VIPs.
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting E-VIPs.
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) setting Virtual Routes
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: (VR_186) Sending/queueing gratuitous ARPs on ha-f3350150-28 for 169.254.0.186
Jan 27 10:09:31 xxxxxxxxx Keepalived_vrrp[1532]: Sending gratuitous ARP on ha-f3350150-28 for 169.254.0.186

And the routing table looks fine:

default via 100.100.100.33 dev qg-7b9963a7-72
169.254.0.0/24 dev ha-a1f79365-fc proto kernel scope link src 169.254.0.186
169.254.192.0/18 dev ha-a1f79365-fc proto kernel scope link src 169.254.192.14
192.168.199.0/24 dev qr-24c07a36-4f proto kernel scope link src 192.168.199.1
100.100.100.32/29 dev qg-7b9963a7-72 scope link
100.100.100.40/29 dev qg-7b9963a7-72 proto kernel scope link src 100.100.100.43

To me, it looks like the order in neutron/agent/linux/keepalived.py has to be changed?

[Florian Engelmann (engelmann)]

Adding "onlink" also solves the issue:

    virtual_routes {
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72 onlink
        100.100.100.32/29 dev qg-7b9963a7-72 scope link
    }

->

Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) Receive advertisement timeout
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) Entering MASTER STATE
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) setting VIPs.
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) setting E-VIPs.
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) setting Virtual Routes
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: (VR_186) Sending/queueing gratuitous ARPs on ha-f3350150-28 for 169.254.0.186
Jan 27 11:16:22 xxxxxxxxx Keepalived_vrrp[1715]: Sending gratuitous ARP on ha-f3350150-28 for 169.254.0.186

->

default via 100.100.100.33 dev qg-7b9963a7-72 proto 18 onlink
169.254.0.0/24 dev ha-f3350150-28 proto kernel scope link src 169.254.0.186
169.254.192.0/18 dev ha-f3350150-28 proto kernel scope link src 169.254.192.2
192.168.199.0/24 dev qr-24c07a36-4f proto kernel scope link src 192.168.199.1
100.100.100.32/29 dev qg-7b9963a7-72 proto 18 scope link
100.100.100.40/29 dev qg-7b9963a7-72 proto kernel scope link src 100.100.100.43

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Florian:

After [1] (Wallaby, Neutron 18), you can't create a subnet with a subnet pool defining the GW IP. Can you print the subnets "show" command? What is the actual GW IP assigned to those subnets?

In any case, you can't have a GW IP outside the subnet range. If, for example, the subnet CIDR is 100.100.100.40/29, the GW IP cannot be 100.100.100.33. GW IP must be in the same subnet CIDR.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/771621

[Florian Engelmann (engelmann)]

Hello Rodolfo,

openstack subnet show xxxxxxxxx -c gateway_ip -c allocation_pools -c cidr
+------------------+-------------------------------+
| Field | Value |
+------------------+-------------------------------+
| allocation_pools | 100.100.100.42-100.100.100.46 |
| cidr | 100.100.100.40/29 |
| gateway_ip | 100.100.100.33 |
+------------------+-------------------------------+

openstack subnet show yyyyyyyyy -c gateway_ip -c allocation_pools -c cidr
+------------------+-------------------------------+
| Field | Value |
+------------------+-------------------------------+
| allocation_pools | 100.100.100.34-100.100.100.38 |
| cidr | 100.100.100.32/29 |
| gateway_ip | 100.100.100.33 |
+------------------+-------------------------------+

You are right, the GW is out of the subnet xxxxxxxxx BUT this is not a problem from a technical point of view (with L3-Agent and Keepalived).
Two options:
A: The Linux kernel is able to route packages to that GW with a "onlink" route.
B: Keepalived (in rocky) is configured (by the l3 agent) to add a net route:

    virtual_routes {
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72
        100.100.100.32/29 dev qg-7b9963a7-72 scope link <<<<<<<<<<<<<
    }

The only problem with option B is the "order". The net route has to exist before the default route is added.

If this is not possible anymore with neutron >= 18.x.x how to do the following:

1. Admin: create a public network
2. Admin: create a subnetpool for this network
3. User: create a subnet from this subnetpool in this public network to get a "dedicated" public IP range that can be used for floating IPs and routers.

When the "physical" network was created, there is only ONE gateway IP. This single GW IP has to be used for each subnet, right?

All the best,
Florian

[Rafael Weingartner (rafaelweingartner)]

Rodolfo,
To build on the, the link you posted, the code that is introduce there does not block the use case described by Florian.

We can see in [1] that the validation performed in [2] would only affect situations where one defines the gateway (GW) of the network as the IP used to address the broadcast or the network itself. Therefore, IPs outside of the CIDR are not validated.

Moreover, the error message displayed in [3] is a little bit misleading. The code is blocking the use of the IP address used to reference the network and broadcast domain, and not IPs that are outside of the CIDR. I will propose a patch to address that.

Moreover, with respect to Florian's use case, there is already code upstream to handle such situations, as one can see in [4]. The that happened with Florian only exists because he is not using upstream Neutron version.

[1] https://github.com/openstack/neutron/blob/e6eeb700c88499281d3924de6d8fd009d5dc09b7/neutron/ipam/utils.py#L37

[2] https://github.com/openstack/neutron/blob/e6eeb700c88499281d3924de6d8fd009d5dc09b7/neutron/ipam/requests.py#L119

[3] https://github.com/openstack/neutron/blob/e6eeb700c88499281d3924de6d8fd009d5dc09b7/neutron/ipam/requests.py#L120

[4] https://github.com/openstack/neutron/blob/e6eeb700c88499281d3924de6d8fd009d5dc09b7/neutron/agent/l3/ha_router.py#L294

[Florian Engelmann (engelmann)]

Hello Rafael,

thank you very much for these details. So we are able to upgrade without regression. That is good news.

We did a hotfix for our outdated Neutron (rocky 13.0.6) version to fix the issue. We will upgrade asap.

Thank you guys for your time and help! I guess we can close this one.

Best regards,
Florian

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Rafael:

Can you please describe why [1] is not preventing from creating a subnet with a netpool and a GW outside this CIDR? Please check [2]. The check "ip in net" will discard the GW outside the subnetpool ("subnet_cidr"). You can try that in the CLI with a couple of commands.

@Floarian, is that correct that you were using code that is not in the upstream reporitory? We must have this information during the code triage and we can't cover this case.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/771621
[2]https://review.opendev.org/c/openstack/neutron/+/771621/8/neutron/ipam/requests.py#119

[Florian Engelmann (engelmann)]

Hello Rodolfo,

the change we did to fix our situation was:

--- a/neutron/agent/linux/keepalived.py
+++ b/neutron/agent/linux/keepalived.py
@@ -144,7 +144,7 @@ class KeepalivedInstanceRoutes(object):

     @property
     def routes(self):
- return self.gateway_routes + self.extra_routes + self.extra_subnets
+ return self.extra_subnets + self.gateway_routes + self.extra_routes

     def __len__(self):
         return len(self.routes)
--
2.34.1

So the extra subnet is set before the default route (with the GW out of CIDR of the "normal" subnet) is set.

Example result:
before:
    virtual_routes {
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72
        100.100.100.32/29 dev qg-7b9963a7-72 scope link
    }

after patch:

    virtual_routes {
        100.100.100.32/29 dev qg-7b9963a7-72 scope link
        0.0.0.0/0 via 100.100.100.33 dev qg-7b9963a7-72
    }

All the best,
Florian

[Florian Engelmann (engelmann)]

I guess an important comment:

Please do not restrict GW IPs to be in the subnet. This would be a horrible regression and destroy a very valid use case.

I am very happy to brainstorm and explain why.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/872219
Committed: https://opendev.org/openstack/neutron/commit/83da1e6d79c4d971718d31cbd3aa038353adbae8
Submitter: "Zuul (22348)"
Branch: master

commit 83da1e6d79c4d971718d31cbd3aa038353adbae8
Author: Rafael Weingärtner <email address hidden>
Date: Mon Jan 30 16:10:26 2023 -0300

    Improve message for subnet gateway out of host IP addresses range

    Following the discussions on #2004004, we notice that there is an error
    message in the Neutron code that is misleading.

    This patch intends to improve the exception message, and provide a
    little bit of documentation regarding the gateway IP address
    validation for subnets.

    Related-bug: #2004004

    Change-Id: Ibf09254b5b2fee6efd3de5e5dc6f013424831db9