neutron

Failed to process compatible router - network is unreachable

Bug #1982882 reported by Adam Gorski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
New
Undecided
Unassigned

Bug Description

I am using kolla-ansible to install OpenStack on 3 servers:

Controller
Compute1
Compute2

Each server has three network interfaces:
1. management (10.0.0.0/24)
2. provider (xxx.xxx.xxx.160/27)
3. server internet access (xxx.xxx.xxx.160/27)

After deployment, I am unable to get internet connectivity on any instance. The following error is seen in /var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I am unsure what changes I can make to resolve this. Network and router information is below.

LOG:

2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting processing update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device qg-6a95a75b-9b already exists
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101, 'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 174, in call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self.process_external()
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "<decorator-gen-92>", line 2, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return f(*a, **k)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self._process_external_gateway(ex_gw_port)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 927, in _process_external_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self.external_gateway_added(ex_gw_port, interface_name)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 880, in external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ex_gw_port, interface_name, self.ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 823, in _external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 853, in _external_gateway_settings
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info device.route.add_gateway(ip)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 613, in add_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info scope=scope)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 651, in add_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info table=table, metric=metric, scope=scope, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 1525, in add_ip_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info metric=metric, scope=scope, proto=proto, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info r_call_timeout)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c: pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')

SUBNETS:

root@deployment:~# openstack subnet list --long
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| ID | Name | Network | Subnet | Project | DHCP | Name Servers | Allocation Pools | Host Routes | IP Version | Gateway | Service Types | Tags |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet | 12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24 | 56648a9748c64ce8a99efab01210ce06 | True | | 10.0.0.2-10.0.0.254 | | 4 | 10.0.0.1 | | |
| fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet | 57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 | 56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 | xxx.xxx.xxx.175-xxx.xxx.xxx.190 | | 4 | xxx.xxx.xxx.161 | | |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+

ROUTER:

root@deployment:~# openstack router list --long
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| ID | Name | Status | State | Project | Distributed | HA | Routes | External gateway info
                                                                                                                         | Availability zones | Tags |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP | 56648a9748c64ce8a99efab01210ce06 | False | False | | {"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips": [{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address": "xxx.xxx.xxx.190"}], "enable_snat": true} | nova | |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+

Thank you in advance for the help.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Adam:

That error happens when the GW IP is not in the same CIDR of the GW port IP address. But in this case seems to be. Can you manually check what is the IP address of device "qg-6a95a75b-9b" (in the router namespace) and the routing table?

Regards.

Revision history for this message
Adam Gorski (thegorski) wrote :

Rodolfo,

Thank you for the response. My provider network interface is eno2 (I assume this is the GW IP you're referring to). On the controller node, the interface details are below. Whenever I run the kolla deployment script the IP of the eno2 interface disappears, which causes internet connectivity loss, which necessitated the addition of the enx interface for internet connectivity.

Since this interface does not have an IP it makes sense that the GW IP is not in the same CIDR of the GW port IP address.

root@controller:/etc/network# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether <mac> brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/24 brd 10.0.0.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet 10.0.0.10/32 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::e42:a1ff:fee1:f938/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether <mac> brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e42:a1ff:fee1:f939/64 scope link
       valid_lft forever preferred_lft forever
4: enx00051bd26dac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether <mac> brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.168/27 brd xxx.xxx.xxx.191 scope global enx00051bd26dac
       valid_lft forever preferred_lft forever
    inet6 xxxx::xxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c2:84:83:f9:b1:0f brd ff:ff:ff:ff:ff:ff
24: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:42:a1:e1:f9:39 brd ff:ff:ff:ff:ff:ff

I notice that the br-ex interface is down and the br-int and br-tun interfaces do not exist. Based on neutron architecture diagrams I've seen I assume this is part of the problem but would also assume these bridges would be created during the deployment process, so I'm not sure how to fix the issue post-deployment.

I am not sure whether this is also relevant but I have enable_neutron_provider_networks = "yes" in my globals.yml file.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Adam:

What you need to check is what is inside the router namespace, not in the root namespace. Also did you notice any *other* L3 agent error, before the one you have pasted?

Regards.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.