TestTimer breaks VPNaaS functional tests

[Bodo Petermann]

Some functional tests for neutron-vpnaas make use of the NamespaceFixture.
If the tests are run in combination with a recent neutron version some tests
fail because the TestTimer raises a TestTimerTimeout even if the namespace
cleanup finishes before the timeout.

In the test setup the tox env for dsvm-functional-sswan will normally
install neutron 17.0.0 (victoria), but for my tests I needed a recent
neutron, so I installed it as an additional step in the setup of the tox env.

The test setup steps are like these, on an Ubuntu 20.04 host:

git clone https://git.openstack.org/openstack-dev/devstack
git clone https://opendev.org/openstack/neutron
git clone https://opendev.org/openstack/neutron-vpnaas
cd neutron-vpnaas
VENV=dsvm-functional-sswan ./tools/configure_for_vpn_func_testing.sh ../devstack -i
tox -e dsvm-functional-sswan --notest
source .tox/dsvm-functional-sswan/bin/activate
python -m pip install ../neutron
deactivate

Then run the neutron-vpnaas functional tests:

tox -e dsvm-functional-sswan

Some tests fail and you see the TestTimerTimeout exception.

The tests were fine with neutron 17.0.0.
The TestTimer was introduced later.
See
Change set https://review.opendev.org/c/openstack/neutron/+/754938/
Related bug https://bugs.launchpad.net/neutron/+bug/1838793

I could narrow the problem with the TestTimer down.
In at least one neutron-vpnaas test
(neutron_vpnaas.tests.functional.strongswan.test_netns_wrapper.TestNetnsWrapper.test_netns_wrap_success)
the NamespaceFixture is used.
The TestTimer is set up, the test completes and the namespace is deleted
successfully before the 5 seconds of the timer are over. But shortly after
that the timer still fires.

The problem is the following: on timer start the old signal handler is
stored (Handler.SIG_DFL in my case) and the remaining time of any existing
alarm (in my case 0). On exit the signal handler is supposed to be reset
and the alarm too. But neither happens.
The signal handler is not set back, because Handler.SIG_DFL is falsy.
The alarm is not stopped because the old value was 0 (there was no ongoing
alarm). So in the end the alarm started by TestTimer will eventually be
signalled.

References:
Change set where the TestTimer was introduced:
https://review.opendev.org/c/openstack/neutron/+/754938/
That related to bug #1838793

[Bodo Petermann (bpetermann)]

Added a fix: https://review.opendev.org/c/openstack/neutron/+/771436

[Bence Romsics (bence-romsics)]

I'm wondering if the root cause is not reinstalling the default signal handler as you write, or that TestTimer is used outside of a Timeout context.

[Bodo Petermann (bpetermann)]

I see. Is there a way to run the tests using the Timeout fixture automatically? Or should the neutron-vpnaas tests be changed to use the Timeout fixture explicitly?

[Bodo Petermann (bpetermann)]

Setting OS_TEST_TIMEOUT in the tox env avoids the problem as TestTimer will find an existing alarm to re-arm in __exit__.
I will prepare a patch for neutron-vpnaas to set OS_TEST_TIMEOUT.

Maybe the patch for TestTimer is still acceptable though as it makes it more robust.

[Bodo Petermann (bpetermann)]

Since the neutron patch is merged and fixes the issue I did not file a patch for neutron-vpnaas to set OS_TEST_TIMEOUT.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.