neutron unit tests break with oslo.utils 3.39.1 and above

Bug #1812922 reported by Nate Johnston on 2019-01-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Unassigned
oslo.utils
Medium
Ben Nemec

Bug Description

The new oslo.service version bump [1] is failing on the Neutron "test_periodic_worker_lifecycle" unit test [2]. This occurs when oslo.utils 3.39.1 or higher is installed; technically 3.38.0 is what is set forth in upper-constraints.txt [3] but you can see oslo.utils 3.40.1 being installed in the failing tests [4].

The change in oslo.config that is precipitating this failure is "Fix race condition in eventletutils Event" [5]. This is now included as a result of the bump for oslo.utils in upper-constraints.txt [6] that occurred yesterday (Jan 21 2019).

As a temporary measure, will alter Neutron requirements.txt to cap the oslo.utils version at 3.39.0 or less so this issue can be fixed.

[1] https://review.openstack.org/632169
[2] neutron.tests.unit.test_worker.PeriodicWorkerTestCase.test_periodic_worker_lifecycle
[3] https://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n548
[4] http://logs.openstack.org/69/632169/1/check/cross-neutron-py35/6f91a77/job-output.txt.gz#_2019-01-22_02_46_08_984507
[5] https://review.openstack.org/#/c/618482/13/oslo_utils/eventletutils.py L160
[6] https://review.openstack.org/632170/

Ben Nemec (bnemec) on 2019-01-22
Changed in oslo.utils:
status: New → Triaged
importance: Undecided → Medium
tags: added: oslo unittest
Slawek Kaplonski (slaweq) wrote :
summary: - neutron functional tests break with oslo.utils 3.39.1 and above
+ neutron unit tests break with oslo.utils 3.39.1 and above
Herve Beraud (herveberaud) wrote :

Seems to work fine to me locally with python 2.7, 3.5, 3.6, 3.7 and oslo.utils 3.40.1

http://paste.openstack.org/show/743089/

Ben Nemec (bnemec) wrote :

I think I see the problem here. It won't reproduce without the new oslo.service release too, which is why it's failing on the constraints bump.

When we fixed the race condition in EventletEvent we added an unconditional send on the old event. If the old event was already sent then that results in the error we're seeing here. We should not re-send the event if it was already sent, much like we did in https://github.com/openstack/oslo.utils/commit/14a53c4d8a9d5605b14a503803859df0d6d4b820

I've got a patch written for this and just need to add tests. Should be up for review shortly.

Fix proposed to branch: master
Review: https://review.openstack.org/632758

Changed in oslo.utils:
assignee: nobody → Ben Nemec (bnemec)
status: Triaged → In Progress
Lajos Katona (lajos-katona) wrote :

Thanks Ben, tomorrow I change the requirements patch (https://review.openstack.org/632658) to blacklist the buggy versions.

Change abandoned by Nate Johnston (<email address hidden>) on branch: master
Review: https://review.openstack.org/632562
Reason: Abandoning in favor of correct measures.

Lajos Katona (lajos-katona) wrote :

Change in requirements (https://review.openstack.org/632658) is on its way.

Reviewed: https://review.openstack.org/632758
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=4eb61941f811a8ab96745a5ea07e6a4a89fe764c
Submitter: Zuul
Branch: master

commit 4eb61941f811a8ab96745a5ea07e6a4a89fe764c
Author: Ben Nemec <email address hidden>
Date: Wed Jan 23 15:13:12 2019 +0000

    Avoid double-setting event

    In commit cc8b51e1e16f6bdc7d6c0e571e2002e70cde098d we added a send
    on the old event when clearing an EventletEvent. However, this was
    done unconditionally, which means if the event was already sent
    then we attempt to send it again. This fails with:

    AssertionError: Trying to re-send() an already-triggered event.

    Similar to 14a53c4d8a9d5605b14a503803859df0d6d4b820, we should check
    if self._set is True and if so then we know that the event was
    already sent and we don't need to do it again.

    Change-Id: I660601383072d11e4a077aada8c1b8c30b9d8d1d
    Closes-Bug: 1812922

Changed in oslo.utils:
status: In Progress → Fix Released

This issue was fixed in the openstack/oslo.utils 3.40.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers