The static route removes the local route from qrouter-namespace

[Piotr Łasak]

The static route removes the local route from qrouter-namespace routing table.

I created two networks with subnet:

First:

#openstack network create Network-1 -f value -c id
784615c9-f9c4-4f4e-87d8-7e5dc76fde5e

#openstack subnet create sub-v4 --network 784615c9-f9c4-4f4e-87d8-7e5dc76fde5e --subnet-range 192.168.0.0/24 -f value -c id
0f1f9cfd-0956-46bd-9bcc-c6eca6fa522b

Second:

#openstack network create Network-2 -f value -c id
041011cc-e254-4326-afa7-9951686610fc

#openstack subnet create sub-v4 --network 041011cc-e254-4326-afa7-9951686610fc --subnet-range 172.16.0.0/24 -f value -c id
3a500e8f-9ef8-48de-a148-623d7fa8e580

Create router:

openstack router create Router-Openstack -f value -c id
3d9e5161-0c19-4bcd-abb8-9350bfcefb56

Added subnet to router:

#openstack router add subnet 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 0f1f9cfd-0956-46bd-9bcc-c6eca6fa522b
#openstack router add subnet 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 3a500e8f-9ef8-48de-a148-623d7fa8e580

I now have a router with two interfaces.

#openstack router show 3d9e5161-0c19-4bcd-abb8-9350bfcefb56
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | AZ1, AZ2, AZ3 |
| created_at | 2022-12-06T16:35:59Z |
| description | |
| external_gateway_info | null |
| flavor_id | None |
| id | 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 |
| interfaces_info | [{"port_id": "10db0b0e-7f54-4927-b9ad-4a69515e069c", "ip_address": "192.168.0.1", "subnet_id": "0f1f9cfd-0956-46bd-9bcc-c6eca6fa522b"}, {"port_id": "99dc5f15-1569-46f3-9d4f-4b907d7ae8fe", "ip_address": "172.16.0.1", "subnet_id": "3a500e8f-9ef8-48de-a148-623d7fa8e580"}] |
| location | Munch({'cloud': '', 'region_name': 'PL', 'zone': None, 'project': Munch({'id': 'afdfb965515f4724a21b6823764asdfg', 'name': 'myproject', 'domain_id': 'default', 'domain_name': None})}) |
| name | Router-Openstack |
| project_id | afdfb965515f4724a21b6823764asdfg |
| revision_number | 8 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2022-12-06T16:37:58Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Routing table from qrouter:

neutron-01:~# ip netns exec qrouter-3d9e5161-0c19-4bcd-abb8-9350bfcefb56 ip route
169.254.0.0/24 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.0.82
169.254.192.0/18 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.193.48
172.16.0.0/24 dev qr-99dc5f15-15 proto kernel scope link src 172.16.0.1
192.168.0.0/24 dev qr-10db0b0e-7f proto kernel scope link src 192.168.0.1

I added new static route.

#openstack router set 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 --route destination=192.168.0.0/24,gateway=172.16.0.10

#openstack router show 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 -c routes
+--------+-----------------------------------------------------+
| Field | Value |
+--------+-----------------------------------------------------+
| routes | destination='192.168.0.0/24', gateway='172.16.0.10' |
+--------+-----------------------------------------------------+

Routing table from qrouter:

neutron-01:~# ip netns exec qrouter-3d9e5161-0c19-4bcd-abb8-9350bfcefb56 ip route
169.254.0.0/24 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.0.82
169.254.192.0/18 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.193.48
172.16.0.0/24 dev qr-99dc5f15-15 proto kernel scope link src 172.16.0.1
192.168.0.0/24 via 172.16.0.10 dev qr-99dc5f15-15 proto 112 <----- my static route
192.168.0.0/24 via 172.16.0.10 dev qr-99dc5f15-15 <----- my static route 2x? why 2x?

In this moment I removed wrong route:

#openstack router remove route 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 --route destination=192.168.0.0/24,gateway=172.16.0.10
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | AZ1, AZ2, AZ3 |
| created_at | 2022-12-06T16:35:59Z |
| description | |
| distributed | False |
| external_gateway_info | null |
| flavor_id | None |
| ha | True |
| id | 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 |
| location | Munch({'cloud': '', 'region_name': 'PL', 'zone': None, 'project': Munch({'id': 'afdfb965515f4724a21b6823764asdfg', 'name': 'myproject', 'domain_id': 'default', 'domain_name': None})}) |
| name | Router-Openstack |
| project_id | afdfb965515f4724a21b6823764asdfg |
| revision_number | 10 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2022-12-06T16:47:58Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Routing table after removing the static route:

neutron-01:~# ip netns exec qrouter-3d9e5161-0c19-4bcd-abb8-9350bfcefb56 ip route
169.254.0.0/24 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.0.82
169.254.192.0/18 dev ha-c5a65a6e-92 proto kernel scope link src 169.254.193.48
172.16.0.0/24 dev qr-99dc5f15-15 proto kernel scope link src 172.16.0.1
192.168.0.0/24 via 172.16.0.10 dev qr-99dc5f15-15 <--------------- the route is still present

#openstack router show 3d9e5161-0c19-4bcd-abb8-9350bfcefb56 -c routes
+--------+-------+
| Field | Value |
+--------+-------+
| routes | |
+--------+-------+

Why is it still showing the wrong entry in the routing table and the local route is missing?
I believe that after deleting the route, the routing entry should look like below:
192.168.0.0/24 dev qr-10db0b0e-7f proto kernel scope link src 192.168.0.1

Does anyone know how to solve this problem?

My software version:

neutron-01:~# dpkg -l | grep neutron | grep ii
ii neutron-bgp-dragent 2:16.0.0-0ubuntu0.20.04.1 all OpenStack Neutron Dynamic Routing - Agent
ii neutron-common 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-dynamic-routing-common 2:16.0.0-0ubuntu0.20.04.1 all OpenStack Neutron Dynamic Routing - common files
ii neutron-fwaas-common 1:16.0.0-0ubuntu0.20.04.1 all Firewall-as-a-Service driver for OpenStack Neutron
ii neutron-l3-agent 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-linuxbridge-agent 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - linuxbridge agent
ii neutron-metadata-agent 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - server
ii neutron-vpnaas-common 2:16.0.0-0ubuntu0.20.04.1 all VPN-as-a-Service driver for OpenStack Neutron
ii python3-neutron 2:16.4.2-0ubuntu2 all Neutron is a virtual network service for Openstack - Python library
ii python3-neutron-dynamic-routing 2:16.0.0-0ubuntu0.20.04.1 all OpenStack Neutron Dynamic Routing - Python 3 library
ii python3-neutron-fwaas 1:16.0.0-0ubuntu0.20.04.1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python3-neutron-lib 2.3.0-0ubuntu1 all Neutron shared routines and utilities - Python 3.x
ii python3-neutron-vpnaas 2:16.0.0-0ubuntu0.20.04.1 all VPN-as-a-Service driver for OpenStack Neutron
ii python3-neutronclient 1:7.1.1-0ubuntu1 all client API library for Neutron - Python 3.x

neutron-01:~# uname -a
Linux neutron-01 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

neutron-01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

[Brian Haley (brian-haley)]

I can confirm that adding/removing a route that is the same as the subnet of the attached router interface can cause issues. I saw something different than described in that I didn't see a duplicate route, but in the end I was left without my interface route.

For example...

# Added second subnet and interface on router, 10.0.0.64/26 here:

default via 172.24.4.1 dev qg-ae7b835f-4a proto static
10.0.0.0/26 dev qr-648ebb56-c3 proto kernel scope link src 10.0.0.1
10.0.0.64/26 dev qr-c052ba70-be proto kernel scope link src 10.0.0.65
172.24.4.0/24 dev qg-ae7b835f-4a proto kernel scope link src 172.24.4.209

# Added route, which removed existing interface route
$ openstack router set router1 --route destination=10.0.0.64/26,gateway=10.0.0.10

default via 172.24.4.1 dev qg-ae7b835f-4a proto static
10.0.0.0/26 dev qr-648ebb56-c3 proto kernel scope link src 10.0.0.1
10.0.0.64/26 via 10.0.0.10 dev qr-648ebb56-c3 proto static
172.24.4.0/24 dev qg-ae7b835f-4a proto kernel scope link src 172.24.4.209

# Removed route, so router still has interface, but no route to the local subnet
$ openstack router remove route router1 --route destination=10.0.0.64/26,gateway=10.0.0.10

default via 172.24.4.1 dev qg-ae7b835f-4a proto static
10.0.0.0/26 dev qr-648ebb56-c3 proto kernel scope link src 10.0.0.1
172.24.4.0/24 dev qg-ae7b835f-4a proto kernel scope link src 172.24.4.209

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I've seen that before in [1][2] and that was documented in [3] (check "important" note). IMO, this behaviour should not be forbidden by the API but at least we should document it, as in [3].

Regards.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1836870
[2]https://bugzilla.redhat.com/show_bug.cgi?id=1836870#c3
[2]https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/networking_guide/index#add-router_common-network-tasks

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

This documentation is present in the neutron-lib API reference: https://docs.openstack.org/api-ref/network/v2/#routers-routers. There is a warning message that warns about the issue described in this bug:

"""
By default in a router there is one route for each attached subnet. If you add an extra route that matches one of the default routes for a subnet, the existing subnet route will be overwritten. If the Neutron route is removed, the corresponding route will be removed as well. The affected subnet will subsequently lose connectivity to this router.
"""

Regards.