vpnaas not working on centos8-stream on xena

[Jacolex]

Hello

After configuring VPN Endpoint, the l3 agent has problem with start the vpn service:

2022-09-02 13:54:02.390 654 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [-] Failed to enable vpn process on router 3659d2d3-5c2e-4097-92dc-08f1567524f5: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-3659d2d3-5c2e-4097-92dc-08f1567524f5', '/var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper', '--mount_paths=/etc:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc,/var/run:/var/lib/neutron/ipsec/3659d2d3-5c2e-409
7-92dc-08f1567524f5/var/run', '--rootwrap_config=/etc/neutron/rootwrap.conf', '--cmd=ipsec,_stackmanager,start']; Stdin: ; Stdout: 2022-09-02 13:54:01.673 88268 INFO neutron.common.config [-] Logging enabled!ESC[00m
2022-09-02 13:54:01.674 88268 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper version 19.3.1.dev44ESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc', '/etc'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:54:01.693 88268 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc has been bind-mounted in /etcESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '/var/run'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:54:01.714 88268 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run has been bind-mounted in /var/runESC[00m
Command: ['ipsec', '_stackmanager', 'start'] Exit code: 1 Stdout: Stderr: cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]

So I did the workaround putting into
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/template/openswan/ipsec.conf.template

config setup
    #nat_traversal=yes

After that the second problem appeared:

2022-09-02 13:41:35.252 35 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [req-aa8d3095-578e-4747-a708-d55d3a4ff889 7a8ec6fc4ec12049bb7f243a354430b4b5ecc5a3fedcdc1c555f1f1a5ce70eb5 715cf7f57a6f47119161fe0654ed8a1c - - -] Failed to enable vpn process on router 3659d2d3-5c2e-4097-92dc-08f1567524f5: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'qrouter-3659d2d3-5c2e-4097-92dc-08f1567524f5', '/var/lib/kolla/venv/bin/neutron-vpn-netns-w
rapper', '--mount_paths=/etc:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc,/var/run:/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '--rootwrap_config=/etc/neutron/rootwrap.conf', '--cmd=ipsec,pluto,--use-netkey,--uniqueids']; Stdin: ; Stdout: 2022-09-02 13:41:34.832 14537 INFO neutron.common.config [-] Logging enabled!ESC[00m
2022-09-02 13:41:34.834 14537 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-vpn-netns-wrapper version 19.3.1.dev44ESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc', '/etc'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:41:34.845 14537 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/etc has been bind-mounted in /etcESC[00m
Command: ['mount', '--bind', '/var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run', '/var/run'] Exit code: 0 Stdout: Stderr: 2022-09-02 13:41:34.856 14537 INFO neutron_vpnaas.services.vpn.common.netns_wrapper [-] /var/lib/neutron/ipsec/3659d2d3-5c2e-4097-92dc-08f1567524f5/var/run has been bind-mounted in /var/runESC[00m
Command: ['ipsec', 'pluto', '--use-netkey', '--uniqueids'] Exit code: 1 Stdout: Stderr: /usr/libexec/ipsec/pluto: unrecognized option '--use-netkey'
For usage information: /usr/libexec/ipsec/pluto --help
Libreswan 4.5

So I deployed the second workaround in
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_vpnaas/services/vpn/device_drivers/libreswan_ipsec.py

    def start_pluto(self):
        cmd = ['pluto',
               '--use-netkey',
               '--uniqueids']

And removed --use-netkey:
    def start_pluto(self):
        cmd = ['pluto',
               '--uniqueids']

After that the vpn endpoint starts working correctly.
Seems there is some problems with libreswan version.
Image version:
quay.io/openstack.kolla/centos-source-neutron-l3-agent
                "build-date": "20220726",

[Jeffrey Zhang (jeffrey4l)]

the same issue with https://bugs.launchpad.net/neutron/+bug/1938571

[Ian Kumlien (pomac)]

So, from what i can see only openswan has support for --use-netkey - which means that if you support free/libre-swan then it should be removed.

[Jacolex (jacolex)]

Based on kolla scripts - libreswan should be supported on centos installation

/usr/local/share/kolla-ansible/ansible/roles/neutron/templates/l3_agent.ini.j2

#jinja2: trim_blocks: False
{% set vpn_device_driver = 'neutron_vpnaas.services.vpn.device_drivers.strongswan_ipsec.StrongSwanDriver' if kolla_base_distro in ['ubuntu', 'debian'] else 'neutron_vpnaas.services.vpn.device_drivers.libreswan_ipsec.LibreSwanDriver'%}

[...]

{% if enable_neutron_vpnaas | bool %}
[ipsec]
enable_detailed_logging = {{ neutron_logging_debug }}

[vpnagent]
vpn_device_driver = {{ vpn_device_driver }}
{% endif %}

[...]

[Ian Kumlien (pomac)]

Ok, so this should work with libreswan but not with freeswan.

Since the official documentation is useless:
https://libreswan.org/man/ipsec_pluto.8.html
(linked from: https://libreswan.org/man/)

Lets use this instead:
https://manpages.debian.org/unstable/libreswan/pluto.8.en.html

Which clearly states that it should be available at least in 4.3 -- if it's not in 4.5 then something has changed... And since it's in mainline and was added 2014-04-15... I don't know.

However is the netkey behavior something that neutron expects or can it just be ignored?

[Ian Kumlien (pomac)]

As a side note:
libreswan-4.5-1.el8.x86_64

ipsec pluto --help |grep netkey |wc -l
0

ipsec --version
Linux Libreswan 4.5 (XFRM) [cut kernel version]

[Ian Kumlien (pomac)]

So a colleague looked around a bit and it will not work with centos-*-stream (both 8 and 9)

It will however work in Ubuntu

[Ian Kumlien (pomac)]

Apparently it's been removed from Ubuntu 22.10 ;)

[Michal Nasiadka (mnasiadka)]

I believe it's a neutron-vpnaas bug, not kolla-ansible bug.

[Lajos Katona (lajos-katona)]

Am I understand right that on some distros has different swan libs like openswan libreswan etc..., and they differ in what options they support?
If my understanding is right, (without looking into the code) it should be handled in the driver itself.

[Ian Kumlien (pomac)]

Some older versions of ipsec had a command you could use, it's gone in the packaged versions of ipsec, so yes, it could be handled - it could also just be removed.

[Ian Kumlien (pomac)]

Also:
nat_traversal

OBSOLETE. Support for NAT Traversal is always enabled.

https://manpages.debian.org/experimental/libreswan/ipsec.conf.5.en.html

Error log:
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Command: ['ipsec', '_stackmanager', 'start'] Exit code: 1 Stdout: Stderr: cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec cannot load config '/etc/ipsec.conf': /etc/ipsec.conf:3: syntax error, unexpected STRING [nat_traversal]
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec unknown stack
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec ; Stderr:
2023-04-17 12:59:39.391 2524 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec