Hi Rodolfo Alonso, Thanks for the reply. --- Openstack version: stable/yoga(Deploy by kolla-ansible) Enable Neutron VPNaaS Neutron server config: - neutron.conf: ```sh [DEFAULT] debug = False log_dir = /var/log/kolla/neutron use_stderr = False bind_host = 10.9.9.215 bind_port = 9696 api_paste_config = /etc/neutron/api-paste.ini api_workers = 5 metadata_workers = 5 rpc_workers = 3 rpc_state_report_workers = 3 metadata_proxy_socket = /var/lib/neutron/kolla/metadata_proxy interface_driver = openvswitch allow_overlapping_ips = true service_plugins = router,vpnaas transport_url = rabbit://openstack:nRlOBOxPApJ7PJ7Az8sMt2Jlxnf0M4GPWdpnj6bE@10.9.9.215:5672,openstack:nRlOBOxPApJ7PJ7Az8sMt2Jlxnf0M4GPWdpnj6bE@10.9.9.216:5672,openstack:nRlOBOxPApJ7PJ7Az8sMt2Jlxnf0M4GPWdpnj6bE@10.9.9.218:5672// ipam_driver = internal rpc_response_timeout = 600 ``` - neutron_vpnaas.conf: ```sh [service_providers] service_provider = VPN:strongswan:neutron_vpnaas.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default ``` Neutron L3 Agent config: - neutron-l3-agent/l3_agent.ini: ```sh [DEFAULT] agent_mode = legacy [agent] extensions = vpnaas [ipsec] enable_detailed_logging = False [vpnagent] vpn_device_driver = neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver [ovs] ovsdb_connection = tcp:127.0.0.1:6640 ovsdb_timeout = 10 ``` Steps: 1. Create subnetpool: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | address_scope_id | None | | created_at | 2023-02-17T11:51:19Z | | default_prefixlen | 24 | | default_quota | None | | description | test | | id | b059c009-7622-4d2e-b89c-b9e08c0f2298 | | ip_version | 4 | | is_default | False | | max_prefixlen | 32 | | min_prefixlen | 8 | | name | b059c009-7622-4d2e-b89c-b9e08c0f2298 | | prefixes | 10.123.0.0/16 | | project_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | | revision_number | 1 | | shared | False | | tags | | | tenant_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | | updated_at | 2023-02-17T11:51:19Z | +-------------------+--------------------------------------+ 2. Create subnet with subnetpool: +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 10.123.1.2-10.123.1.254 | | cidr | 10.123.1.0/24 | | created_at | 2023-02-17T11:52:04Z | | description | test | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | True | | gateway_ip | 10.123.1.1 | | host_routes | | | id | 5901541e-cf9e-41d4-8336-c6e0a4471848 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | 5901541e-cf9e-41d4-8336-c6e0a4471848 | | network_id | d55f4da4-78e8-424d-bcb4-7b70ba97475e | | project_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | | revision_number | 1 | | segment_id | None | | service_types | | | subnetpool_id | b059c009-7622-4d2e-b89c-b9e08c0f2298 | | tags | | | tenant_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | | updated_at | 2023-02-17T11:52:04Z | +----------------------+--------------------------------------+ 3. Create a router and attach the above subnet to the router: ```sh admin_state_up: UP availability_zones: nova distributed: False external_gateway_info: {"network_id": "95a02662-5bbb-456c-a8a1-16666d8e018d", "external_fixed_ips": [{"subnet_id": "5cfca450-3503-469f-a437-bfbb6a966545", "ip_address": "171.10.1.160"}], "enable_snat": false} interfaces_info: [{"port_id": "7cc2ae88-8f8d-4fc9-80a6-2803ad7a26c9", "ip_address": "10.123.1.4", "subnet_id": "5901541e-cf9e-41d4-8336-c6e0a4471848"}] ``` 4. Create VPN service with above router: +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | Description | | | Flavor | None | | ID | 3f6f3f32-b5d1-4122-ae1c-257ed4b1392c | | Name | vpngateway-new-3 | | Project | 973b5b62-c161-4d52-a79e-c68461debfa8 | | Router | 81bfaaf1-f1ac-4a9c-a63b-0b16f76013f7 | | State | True | | Status | PENDING_CREATE | | Subnet | 5901541e-cf9e-41d4-8336-c6e0a4471848 | | external_v4_ip | 171.10.1.160 | | external_v6_ip | None | | project_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | +----------------+--------------------------------------+ 5. Create IPSec site connection from above VPN service: +--------------------------+----------------------------------------------------+ | Field | Value | +--------------------------+----------------------------------------------------+ | Authentication Algorithm | psk | | Description | | | ID | 024cf284-9cde-4a94-ba09-d8a35cae1064 | | IKE Policy | 7465d25c-9086-4fd2-853c-f63d3f821d82 | | IPSec Policy | acba1956-eea0-4894-9702-9fccde05f047 | | Initiator | bi-directional | | Local Endpoint Group ID | None | | Local ID | | | MTU | 1500 | | Name | datvt | | Peer Address | 140.82.121.3 | | Peer CIDRs | 10.2.2.0/24 | | Peer Endpoint Group ID | None | | Peer ID | 140.82.121.3 | | Pre-shared Key | 4563567 | | Project | 973b5b62-c161-4d52-a79e-c68461debfa8 | | Route Mode | static | | State | True | | Status | PENDING_CREATE | | VPN Service | 3f6f3f32-b5d1-4122-ae1c-257ed4b1392c | | dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} | | project_id | 973b5b62-c161-4d52-a79e-c68461debfa8 | +--------------------------+----------------------------------------------------+ IPSec site connection keep Pending Create status for long time. So I checked neutron-server and got this log: ```sh 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server [req-9d3ddb1f-19e6-466a-8827-4d4720c3f159 e9ad44c4d8bc4a1ca17fd38e44161b29 a2439087-60c0-4939-b381-209d7342ed37 - - -] Can not send reply for message: AttributeError: 'SubnetPool' object has no attribute 'shared' 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 184, in _process_incoming 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server message.reply(res) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 150, in reply 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server self._send_reply(conn, reply, failure) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 125, in _send_reply 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg)) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/common.py", line 292, in serialize_msg 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server _MESSAGE_KEY: jsonutils.dumps(raw_msg)} 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_serialization/jsonutils.py", line 202, in dumps 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return json.dumps(obj, default=default, **kwargs) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return cls( 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/json/encoder.py", line 199, in encode 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server chunks = self.iterencode(o, _one_shot=True) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return _iterencode(o, 0) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_serialization/jsonutils.py", line 161, in to_primitive 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return recursive(dict(value.iteritems()), level=level + 1) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/sqlalchemy/models.py", line 99, in iteritems 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return self._as_dict().items() 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/sqlalchemy/models.py", line 91, in _as_dict 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server local = dict((key, value) for key, value in self) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_db/sqlalchemy/models.py", line 91, in 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server local = dict((key, value) for key, value in self) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/db/model_base.py", line 92, in next 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server return n, getattr(self, n) 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server AttributeError: 'SubnetPool' object has no attribute 'shared' 2023-02-21 10:09:30.301 7 ERROR oslo_messaging.rpc.server ``` And got neutron-l3-agent log below: ```sh 2023-02-21 10:01:45.537 21 ERROR neutron_lib.rpc [req-16e3231d-bda8-43dc-8261-b5686c34240c e9ad44c4d8bc4a1ca17fd38e44161b29 a2439087-60c0-4939-b381-209d7342ed37 - - -] Timeout in RPC method get_vpn_services_on_host. Waiting for 464 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6912b28b1089486688270922dd899362 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server [req-16e3231d-bda8-43dc-8261-b5686c34240c e9ad44c4d8bc4a1ca17fd38e44161b29 a2439087-60c0-4939-b381-209d7342ed37 - - -] Exception during message handling: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6912b28b1089486688270922dd899362 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self._queues[msg_id].get(block=True, timeout=timeout) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return waiter.wait() 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return get_hub().switch() 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self.greenlet.switch() 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server _queue.Empty 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_log/helpers.py", line 67, in wrapper 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 958, in vpnservice_updated 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server self.sync(context, [router] if router else []) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_log/helpers.py", line 67, in wrapper 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 391, in inner 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 1122, in sync 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server vpnservices = self.agent_rpc.get_vpn_services_on_host( 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_log/helpers.py", line 67, in wrapper 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 799, in get_vpn_services_on_host 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return cctxt.call(context, 'get_vpn_services_on_host', host=host) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/rpc.py", line 180, in call 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server time.sleep(wait) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server raise self.value 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/rpc.py", line 157, in call 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self._original_context.call(ctxt, method, **kwargs) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server result = self.transport._send( 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message, 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout, 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server result = self._waiter.wait(msg_id, timeout, 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server message = self.waiters.get(msg_id, timeout=timeout) 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server raise oslo_messaging.MessagingTimeout( 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 6912b28b1089486688270922dd899362 2023-02-21 10:09:29.998 21 ERROR oslo_messaging.rpc.server ``` Best Regards.