VPNaaS: creating ipsec site connection fails (in master)

[Bodo Petermann]

With the current master branch of neutron and neutron-vpnaas requests to create VPN ipsec site connections fail with a 500 Internal Server Error. I noticed that in a devstack with vpnaas enabled.

The automated tests (neutron-tempest-plugin-vpnaas) also started to fail.
See https://zuul.opendev.org/t/openstack/build/7f680cca45544cec9e3418bda7399324

In the neutron-server log there is an exception (see below).

This seems to be related to neutron commit 6b529ea3c559ed1a016e2aa2c30769429192d528
"[sqlalchemy-20] Use session.transaction information to decide if active"
https://review.opendev.org/c/openstack/neutron/+/843256

Probably the neutron-vpnaas code needs to be adapted to avoid the error of remove_reservation inside a transaction.

Exception log:

Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] create failed: No details.: RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource result = method(request=request, **args)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource context_reference.session.rollback()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise()
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return notify({self._resource: self._view(request.context,
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 504, in notify
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota.QUOTAS.commit_reservation(
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/quota/__init__.py", line 103, in commit_reservation
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.get_driver().commit_reservation(context, reservation_id)
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/quota/driver.py", line 271, in commit_reservation
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota_api.remove_reservation(context, reservation_id,
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/common/utils.py", line 700, in inner
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise RuntimeError(_("Method %s cannot be called within a "
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction.
Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource
Jun 12 08:46:58.912701 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: INFO neutron.wsgi [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] 149.202.162.36,149.202.162.36 "POST /v2.0/vpn/ipsec-site-connections HTTP/1.1" status: 500 len: 368 time: 0.2223299

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-vpnaas/+/845759

[Bodo Petermann (bpetermann)]

This patch doesn't work (it breaks creating vpn services). I'm trying another approach.

[Bodo Petermann (bpetermann)]

I updated the patch.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-vpnaas/+/845759
Committed: https://opendev.org/openstack/neutron-vpnaas/commit/740bf12dfa6850c84a3ce07c1a840ae113045f71
Submitter: "Zuul (22348)"
Branch: master

commit 740bf12dfa6850c84a3ce07c1a840ae113045f71
Author: Bodo Petermann <email address hidden>
Date: Tue Jun 14 15:44:32 2022 +0200

    Fix creating ipsec site connection

    Fixes a RuntimeError "Method remove_reservation cannot be called
    within a transaction" while creating an ipsec site connection.
    To avoid the error the service driver now gets the vpnservice
    (to get the router_id) inside a CONTEXT_READER wrapper.

    Closes-Bug: #1978571
    Change-Id: I6552816db8a596a933365dc9de481d8c32371f1c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-vpnaas 21.0.0.0rc1 release candidate.