neutron unit tests break with oslo.utils 3.39.1 and above

[Nate Johnston]

The new oslo.service version bump [1] is failing on the Neutron "test_periodic_worker_lifecycle" unit test [2]. This occurs when oslo.utils 3.39.1 or higher is installed; technically 3.38.0 is what is set forth in upper-constraints.txt [3] but you can see oslo.utils 3.40.1 being installed in the failing tests [4].

The change in oslo.config that is precipitating this failure is "Fix race condition in eventletutils Event" [5]. This is now included as a result of the bump for oslo.utils in upper-constraints.txt [6] that occurred yesterday (Jan 21 2019).

As a temporary measure, will alter Neutron requirements.txt to cap the oslo.utils version at 3.39.0 or less so this issue can be fixed.

[1] https://review.openstack.org/632169
[2] neutron.tests.unit.test_worker.PeriodicWorkerTestCase.test_periodic_worker_lifecycle
[3] https://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n548
[4] http://logs.openstack.org/69/632169/1/check/cross-neutron-py35/6f91a77/job-output.txt.gz#_2019-01-22_02_46_08_984507
[5] https://review.openstack.org/#/c/618482/13/oslo_utils/eventletutils.py L160
[6] https://review.openstack.org/632170/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/632562

[Lajos Katona (lajos-katona)]

I think this is the relevant logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20Trying%20to%20re-send()%20an%20already-triggered%20event%5C%22%20AND%20project%3A%5C%22openstack%2Fneutron%5C%22%20AND%20build_status%3A%5C%22FAILURE%5C%22%20AND%20build_name%3A%5C%22openstack-tox-py27-with-oslo-master%5C%22

[Slawek Kaplonski (slaweq)]

It is failing in periodic jobs which results can be found in http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master/
See Grafana: http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1&panelId=38&fullscreen&from=1547623152690&to=1548227952690

[Herve Beraud (herveberaud)]

Seems to work fine to me locally with python 2.7, 3.5, 3.6, 3.7 and oslo.utils 3.40.1

http://paste.openstack.org/show/743089/

[Ben Nemec (bnemec)]

I think I see the problem here. It won't reproduce without the new oslo.service release too, which is why it's failing on the constraints bump.

When we fixed the race condition in EventletEvent we added an unconditional send on the old event. If the old event was already sent then that results in the error we're seeing here. We should not re-send the event if it was already sent, much like we did in https://github.com/openstack/oslo.utils/commit/14a53c4d8a9d5605b14a503803859df0d6d4b820

I've got a patch written for this and just need to add tests. Should be up for review shortly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/632758

[Lajos Katona (lajos-katona)]

Thanks Ben, tomorrow I change the requirements patch (https://review.openstack.org/632658) to blacklist the buggy versions.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Nate Johnston (<email address hidden>) on branch: master
Review: https://review.openstack.org/632562
Reason: Abandoning in favor of correct measures.

[Lajos Katona (lajos-katona)]

Change in requirements (https://review.openstack.org/632658) is on its way.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/632758
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=4eb61941f811a8ab96745a5ea07e6a4a89fe764c
Submitter: Zuul
Branch: master

commit 4eb61941f811a8ab96745a5ea07e6a4a89fe764c
Author: Ben Nemec <email address hidden>
Date: Wed Jan 23 15:13:12 2019 +0000

    Avoid double-setting event

    In commit cc8b51e1e16f6bdc7d6c0e571e2002e70cde098d we added a send
    on the old event when clearing an EventletEvent. However, this was
    done unconditionally, which means if the event was already sent
    then we attempt to send it again. This fails with:

    AssertionError: Trying to re-send() an already-triggered event.

    Similar to 14a53c4d8a9d5605b14a503803859df0d6d4b820, we should check
    if self._set is True and if so then we know that the event was
    already sent and we don't need to do it again.

    Change-Id: I660601383072d11e4a077aada8c1b8c30b9d8d1d
    Closes-Bug: 1812922

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.utils 3.40.2 release.