I am using kolla-ansible to install OpenStack on 3 servers:
Controller
Compute1
Compute2
Each server has three network interfaces:
1. management (10.0.0.0/24)
2. provider (xxx.xxx.xxx.160/27)
3. server internet access (xxx.xxx.xxx.160/27)
After deployment, I am unable to get internet connectivity on any instance. The following error is seen in /var/log/kolla/neutron/neutron-l3-agent.log after instance creation. I am unsure what changes I can make to resolve this. Network and router information is below.
LOG:
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting processing update 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.061 18 INFO neutron.agent.l3.agent [-] Starting router update for 4e471427-5bf3-4e72-9e56-cea3c8474b0c, action 3, priority 2, update_id 0f830a94-0c6f-4dff-9063-aabad002173a. Wait time elapsed: 6.087
2022-07-26 18:29:26.492 18 INFO neutron.agent.linux.interface [-] Device qg-6a95a75b-9b already exists
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info [-] (101, 'Network is unreachable'): pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 174, in call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self.process_external()
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "<decorator-gen-92>", line 2, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info return f(*a, **k)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self._process_external_gateway(ex_gw_port)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 927, in _process_external_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info self.external_gateway_added(ex_gw_port, interface_name)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 880, in external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ex_gw_port, interface_name, self.ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 823, in _external_gateway_added
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info ns_name, preserve_ips)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 853, in _external_gateway_settings
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info device.route.add_gateway(ip)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 613, in add_gateway
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info scope=scope)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 651, in add_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info table=table, metric=metric, scope=scope, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 1525, in add_ip_route
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info metric=metric, scope=scope, proto=proto, **kwargs)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info r_call_timeout)
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
2022-07-26 18:29:26.976 18 ERROR neutron.agent.l3.router_info
2022-07-26 18:29:26.977 18 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 4e471427-5bf3-4e72-9e56-cea3c8474b0c: pr2modules.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')
SUBNETS:
root@deployment:~# openstack subnet list --long
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| ID | Name | Network | Subnet | Project | DHCP | Name Servers | Allocation Pools | Host Routes | IP Version | Gateway | Service Types | Tags |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
| 76d3d8e6-93c4-407a-af8a-b67c06316644 | private-subnet | 12cd0385-f70d-45f6-afcf-4c835d4039ed | 10.0.0.0/24 | 56648a9748c64ce8a99efab01210ce06 | True | | 10.0.0.2-10.0.0.254 | | 4 | 10.0.0.1 | | |
| fabe73a6-f17e-437c-96d4-59caa9a7e7f3 | public-subnet | 57bb06e9-a4c1-47b8-94de-a5504d910798 | xxx.xxx.xxx.160/27 | 56648a9748c64ce8a99efab01210ce06 | True | xxx.xxx.xxx.22 | xxx.xxx.xxx.175-xxx.xxx.xxx.190 | | 4 | xxx.xxx.xxx.161 | | |
+--------------------------------------+------------------+--------------------------------------+------------------+----------------------------------+------+--------------+-----------------------------+-------------+------------+---------------+---------------+------+
ROUTER:
root@deployment:~# openstack router list --long
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| ID | Name | Status | State | Project | Distributed | HA | Routes | External gateway info
| Availability zones | Tags |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
| 21ecb828-a0cf-4e8e-a500-9395222efc4b | private-ns-router | ACTIVE | UP | 56648a9748c64ce8a99efab01210ce06 | False | False | | {"network_id": "57bb06e9-a4c1-47b8-94de-a5504d910798", "external_fixed_ips": [{"subnet_id": "fabe73a6-f17e-437c-96d4-59caa9a7e7f3", "ip_address": "xxx.xxx.xxx.190"}], "enable_snat": true} | nova | |
+--------------------------------------+-------------------+--------+-------+----------------------------------+-------------+-------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+------+
Thank you in advance for the help.
Hello Adam:
That error happens when the GW IP is not in the same CIDR of the GW port IP address. But in this case seems to be. Can you manually check what is the IP address of device "qg-6a95a75b-9b" (in the router namespace) and the routing table?
Regards.