[SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError

Bug #1863021 reported by Corey Bryant on 2020-02-12
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
BaGPipe
Undecided
Unassigned
Cinder
Undecided
Corey Bryant
Designate
Undecided
Corey Bryant
Glance
Undecided
Unassigned
Mellanox backend integration with Neutron (networking-mlnx)
Undecided
Unassigned
Mistral
Undecided
Unassigned
Murano
Undecided
Corey Bryant
OpenStack Compute (nova)
Undecided
Corey Bryant
OpenStack DBaaS (Trove)
Fix Released
Undecided
Lingxian Kong
OpenStack Object Storage (swift)
Undecided
Chris MacNaughton
OpenStack Shared File Systems Service (Manila)
Undecided
suzhengwei
masakari
Undecided
Corey Bryant
networking-hyperv
Undecided
Corey Bryant
networking-l2gw
Undecided
Unassigned
networking-sfc
Undecided
Unassigned
neutron
High
Dr. Jens Harbott
oslo.service
Undecided
Corey Bryant
senlin
Undecided
Unassigned
watcher
Undecided
Chris MacNaughton
barbican (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Arthur Dayne
cinder (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
designate (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
glance (Ubuntu)
Status tracked in Groovy
Focal
Undecided
Unassigned
Groovy
High
Unassigned
heat (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
ironic (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Arthur Dayne
ironic-inspector (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
magnum (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
manila (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Arthur Dayne
masakari (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
mistral (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Arthur Dayne
murano (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
murano-agent (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
networking-bagpipe (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
networking-hyperv (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
networking-l2gw (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
networking-mlnx (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
networking-sfc (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
neutron (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
neutron-dynamic-routing (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
nova (Ubuntu)
Status tracked in Groovy
Focal
Undecided
Unassigned
Groovy
High
Unassigned
openstack-trove (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
python-os-ken (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
python-oslo.service (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
sahara (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
senlin (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Unassigned
swift (Ubuntu)
Status tracked in Groovy
Focal
Low
Unassigned
Groovy
Low
Unassigned
watcher (Ubuntu)
Status tracked in Groovy
Focal
High
Unassigned
Groovy
High
Arthur Dayne

Bug Description

[Impact]
This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592

However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:

 Exception ignored in: <function _after_fork at 0x7f1100d075e0>
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
     assert len(_active) == 1
 AssertionError:
 Exception ignored in: <function _after_fork at 0x7f1100d075e0>
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
     assert len(_active) == 1
 AssertionError:

Patching nova/monkey_patch.py with the following appears to fix this:

diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py
index a07ff91dac..bb7252c643 100644
--- a/nova/monkey_patch.py
+++ b/nova/monkey_patch.py
@@ -59,6 +59,9 @@ def _monkey_patch():
     else:
         eventlet.monkey_patch()

+ import __original_module_threading
+ import threading
+ __original_module_threading.current_thread.__globals__['_active'] = threading._active
     # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
     # hub use a monotonic clock to avoid issues with drifts of system time (see

Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance.

[Test Case]
Tempest regression on a deployed openstack should pass successfully.

Individual services can be tested as follows but with the scope of this issue tempest regression testing will be the goal for verifying this SRU.
'systemctl status <service>' should not show the failure reported in

[Impact].

[Regression Potential]
Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.

Corey Bryant (corey.bryant) wrote :

See attached for much more detailed recreation/patching/fixing.

tags: added: py38
Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.opendev.org/707474

Changed in nova:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress

This bug was fixed in the package nova - 2:21.0.0~b2~git2020021008.1fcd74730d-0ubuntu2

---------------
nova (2:21.0.0~b2~git2020021008.1fcd74730d-0ubuntu2) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread-active.patch: Cherry-picked from
    https://review.opendev.org/#/c/707474/. This fixes nova service failures
    that autopkgtests are hitting with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 20 Feb 2020 09:35:53 -0500

Changed in nova (Ubuntu):
status: Triaged → Fix Released

Reviewed: https://review.opendev.org/707474
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2078ef850c93b47ec1bcd044450b589d7a8fc0b5
Submitter: Zuul
Branch: master

commit 2078ef850c93b47ec1bcd044450b589d7a8fc0b5
Author: Corey Bryant <email address hidden>
Date: Wed Feb 12 16:32:40 2020 -0500

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I4872169413f27aeaff8d8fdfa5cdaf6ee32f4680
    Closes-Bug: #1863021

Changed in nova:
status: In Progress → Fix Released
Changed in glance (Ubuntu):
status: New → Triaged
importance: Undecided → High
description: updated
description: updated
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glance - 2:20.0.0~b3~git2020032414.30ece7aa-0ubuntu2

---------------
glance (2:20.0.0~b3~git2020032414.30ece7aa-0ubuntu2) focal; urgency=medium

  * d/tests/test-shebang.py: Dropped. No longer used.
  * d/p/monkey-patch-original-current-thread-active.patch: Cherry-picked from
    https://review.opendev.org/#/c/716058/. This fixes glance service failures
    that autopkgtests are hitting with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Mon, 30 Mar 2020 14:41:49 -0400

Changed in glance (Ubuntu):
status: Triaged → Fix Released

Reviewed: https://review.opendev.org/716058
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=90146b62c765ad1b8be1ffec1799cba9f3994c2d
Submitter: Zuul
Branch: master

commit 90146b62c765ad1b8be1ffec1799cba9f3994c2d
Author: Corey Bryant <email address hidden>
Date: Mon Mar 30 15:14:15 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I95a8d8cf02a0cb923418c0b5655442b8d7bc6b08
    Closes-Bug: #1863021

Changed in glance:
status: In Progress → Fix Released

This issue was fixed in the openstack/glance 20.0.0.0b3 development milestone.

description: updated
Changed in cinder (Ubuntu):
status: New → Triaged
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.opendev.org/724753

Changed in neutron:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in cinder:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in neutron (Ubuntu):
importance: Undecided → High
status: New → Triaged
summary: - eventlet monkey patch results in assert len(_active) == 1 AssertionError
+ [SRU] eventlet monkey patch results in assert len(_active) == 1
+ AssertionError
description: updated
Changed in neutron (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in cinder (Ubuntu Focal):
status: New → Fix Released
importance: Undecided → High
status: Fix Released → Triaged
Changed in glance (Ubuntu Focal):
status: New → Fix Released
Changed in nova (Ubuntu Focal):
status: New → Fix Released
Corey Bryant (corey.bryant) wrote :

It looks like at least the following are also affected:

barbican
designate
heat
ironic
ironic-inspector
magnum
manila
masakari
masakari-monitors
mistral
murano
murano-agent
networking-bagpipe
networking-hyperv,
networking-l2gw
networking-mlnx
networking-ovn
networking-sfc
neutron-dynamic-routing
neutron-fwaas
openstack-trove
python-os-ken
python-oslo.concurrency
python-oslo.messaging
python-oslo.service
ryu
sahara
senlin
swift
vmware-nsx
watcher

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu3

---------------
neutron (2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu3) groovy; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/724753. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 30 Apr 2020 16:42:56 -0400

Changed in neutron (Ubuntu Groovy):
status: Triaged → Fix Released
Akihiro Motoki (amotoki) on 2020-05-04
Changed in neutron:
importance: Undecided → High
Changed in heat (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in heat (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in swift (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in swift (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2:16.0.0~b3~git2020041012.eb915e2db-0ubuntu2

---------------
cinder (2:16.0.0~b3~git2020041012.eb915e2db-0ubuntu2) groovy; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/724754. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

 -- Corey Bryant <email address hidden> Thu, 30 Apr 2020 16:44:03 -0400

Changed in cinder (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in designate (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in designate (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725346

Changed in designate:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in barbican (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in barbican (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in ironic (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in ironic (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in ironic-inspector (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in ironic-inspector (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in python-oslo.service (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in python-oslo.service (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725359

Changed in oslo.service:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in networking-bagpipe (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-bagpipe (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725362

Changed in networking-hyperv:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in networking-l2gw (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-l2gw (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-hyperv (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-hyperv (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-mlnx (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in networking-mlnx (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-sfc (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in magnum (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in magnum (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in manila (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725393

Changed in manila:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in manila (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in masakari (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725397

Changed in masakari:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
Changed in masakari (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in mistral (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in mistral (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in neutron-dynamic-routing (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in neutron-dynamic-routing (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in neutron:
assignee: Corey Bryant (corey.bryant) → Dr. Jens Harbott (j-harbott)

Reviewed: https://review.opendev.org/725359
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=b83ce338312bfeabda6b6755f49530acde61141e
Submitter: Zuul
Branch: master

commit b83ce338312bfeabda6b6755f49530acde61141e
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:06:38 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Icc2277b72f6f8f7812be22c43bbc281334aa2373
    Closes-Bug: #1863021

Changed in oslo.service:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/725403
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=44e77ea9a975e9222f38591151e2c32640ef7379
Submitter: Zuul
Branch: master

commit 44e77ea9a975e9222f38591151e2c32640ef7379
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:31:09 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I8e8a9ce5085b7915994317af5ea903cf8a39e809
    Closes-Bug: #1863021

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/724753
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd
Submitter: Zuul
Branch: master

commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021

Changed in murano (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in murano (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in murano-agent (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in murano-agent (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in openstack-trove (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in openstack-trove (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in sahara (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in sahara (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in senlin (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in senlin (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-sfc (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged

Fix proposed to branch: master
Review: https://review.opendev.org/725670

Changed in murano:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress

Fix proposed to branch: master
Review: https://review.opendev.org/725671

Changed in trove:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress

Reviewed: https://review.opendev.org/724754
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9f0aae5b5876b6a85fc7a30a4acb1f33f76fdb03
Submitter: Zuul
Branch: master

commit 9f0aae5b5876b6a85fc7a30a4acb1f33f76fdb03
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:31:18 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Ida548b4bec00530418fd3d7ab254e971af77d3fe
    Closes-Bug: #1863021

Changed in cinder:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/725671
Committed: https://git.openstack.org/cgit/openstack/murano-agent/commit/?id=f2e41f2f659fc4490e55d3372cfbcebb295c8ad4
Submitter: Zuul
Branch: master

commit f2e41f2f659fc4490e55d3372cfbcebb295c8ad4
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:41:02 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
    Closes-Bug: #1863021

Changed in murano:
status: In Progress → Fix Released
Changed in swift:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Changed in watcher:
assignee: nobody → Chris MacNaughton (chris.macnaughton)
status: New → In Progress

Reviewed: https://review.opendev.org/725670
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=fe543a67c70e212967bda0e3e45dc9363eaf8fcf
Submitter: Zuul
Branch: master

commit fe543a67c70e212967bda0e3e45dc9363eaf8fcf
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:39:52 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Also disable E402 to allow the change in.

    Change-Id: I508fcd0707ecdd2bf720303f6cbb4087a38aaadd
    Closes-Bug: #1863021

Reviewed: https://review.opendev.org/725639
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e81440b63dcafee9ad693dc0198c1598631086d
Submitter: Zuul
Branch: stable/ussuri

commit 1e81440b63dcafee9ad693dc0198c1598631086d
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021
    (cherry picked from commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd)

tags: added: in-stable-ussuri

Reviewed: https://review.opendev.org/725774
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9cae1e4660647dc06e1554b4fb74ea75abdad19a
Submitter: Zuul
Branch: stable/train

commit 9cae1e4660647dc06e1554b4fb74ea75abdad19a
Author: Corey Bryant <email address hidden>
Date: Thu Apr 30 15:44:10 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I49bfd9673abc7602b27dc48b8b490daaded2882c
    Closes-Bug: #1863021
    (cherry picked from commit ec7a5aa5161f438736c5a1dcf10f5d0e48f329cd)

tags: added: in-stable-train

Reviewed: https://review.opendev.org/725796
Committed: https://git.openstack.org/cgit/openstack/murano-agent/commit/?id=97f53bb54a6209bc917e1a5a727ea8f6333bafc1
Submitter: Zuul
Branch: stable/ussuri

commit 97f53bb54a6209bc917e1a5a727ea8f6333bafc1
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:41:02 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
    Closes-Bug: #1863021
    (cherry picked from commit f2e41f2f659fc4490e55d3372cfbcebb295c8ad4)

Changed in networking-l2gw:
status: In Progress → Fix Released
Changed in networking-bagpipe:
status: In Progress → Fix Released
Changed in networking-sfc:
status: In Progress → Fix Released
Changed in networking-hyperv:
status: In Progress → Fix Released
Changed in python-os-ken (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in python-os-ken (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in watcher (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in watcher (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Arthur Dayne (palagend) on 2020-05-08
Changed in barbican (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in mistral (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in ironic (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Arthur Dayne (palagend) on 2020-05-08
Changed in manila (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in watcher (Ubuntu Groovy):
assignee: nobody → Arthur Dayne (palagend)
Changed in manila:
assignee: Corey Bryant (corey.bryant) → suzhengwei (sue.sam)
Changed in senlin:
status: In Progress → Fix Released
Changed in swift (Ubuntu Focal):
importance: High → Low
Changed in swift (Ubuntu Groovy):
importance: High → Low
Changed in ironic (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in mistral:
status: In Progress → Fix Released
Changed in watcher:
status: In Progress → Fix Released
Changed in designate (Ubuntu Groovy):
status: Triaged → Fix Released
description: updated
Changed in barbican (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-hyperv (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-bagpipe (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in ironic-inspector (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in manila (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in python-os-ken (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-sfc (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in python-oslo.service (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-mlnx (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in mistral (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in senlin (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in watcher (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in openstack-trove (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in murano (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in murano-agent (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in networking-l2gw (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in neutron-dynamic-routing (Ubuntu Groovy):
status: Triaged → Fix Released
Steve Langasek (vorlon) on 2020-05-15
Changed in ironic (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Changed in magnum (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in manila:
status: In Progress → Fix Released
Steve Langasek (vorlon) on 2020-05-15
Changed in heat (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon) on 2020-05-15
Changed in designate (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon) on 2020-05-15
Changed in barbican (Ubuntu Focal):
status: Triaged → Fix Committed
Steve Langasek (vorlon) on 2020-05-15
Changed in ironic-inspector (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in magnum (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in openstack-trove (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in masakari (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-mlnx (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-bagpipe (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in python-os-ken (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in python-oslo.service (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-hyperv (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-sfc (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in manila (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in mistral (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in senlin (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in masakari (Ubuntu Focal):
status: Fix Committed → Triaged
62 comments hidden view all 142 comments

Hello Corey, or anyone else affected,

Accepted watcher into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/watcher/1:4.0.0-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in watcher (Ubuntu Focal):
status: Triaged → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package masakari - 9.0.0-0ubuntu3

---------------
masakari (9.0.0-0ubuntu3) groovy; urgency=medium

  * d/control, d/py3dist-overrides: Drop (Build-)Depends on
    universe package python3-sqlalchemy-utils. This was added in
    9.0.0 but only needed for upstream devstack gate job.

 -- Corey Bryant <email address hidden> Tue, 19 May 2020 13:00:30 -0400

Changed in masakari (Ubuntu Groovy):
status: Triaged → Fix Released
Brian Murray (brian-murray) wrote :

Hello Corey, or anyone else affected,

Accepted masakari into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/masakari/9.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in masakari (Ubuntu Focal):
status: Triaged → Fix Committed

Reviewed: https://review.opendev.org/728397
Committed: https://git.openstack.org/cgit/openstack/watcher/commit/?id=043ebb71c6354b62b56b0361b7fdbff54146241e
Submitter: Zuul
Branch: stable/ussuri

commit 043ebb71c6354b62b56b0361b7fdbff54146241e
Author: Chris MacNaughton <email address hidden>
Date: Wed May 6 08:10:32 2020 +0200

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I194eedd505d45137963eb40d1b1d5da2309caeac
    Closes-Bug: #1863021
    (cherry picked from commit 0b4c4f1de68631164d62dbf66160e5d8cb923518)

Changed in trove:
assignee: Corey Bryant (corey.bryant) → Lingxian Kong (kong)
Corey Bryant (corey.bryant) wrote :

Regression testing on focal-proposed was successful. Tempest results:

======
Totals
======
Ran: 92 tests in 1691.7210 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 684.1236 sec.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Changed in cinder (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in murano (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in murano-agent (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in networking-l2gw (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in neutron (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in neutron-dynamic-routing (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in sahara (Ubuntu Focal):
status: Triaged → Fix Committed
Changed in sahara (Ubuntu Groovy):
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package heat - 1:14.0.0-0ubuntu1

---------------
heat (1:14.0.0-0ubuntu1) groovy; urgency=medium

  * d/watch: Scope to 14.x series and get tarballs from opendev.org.
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/727181/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * New upstream release for OpenStack Ussuri (LP: #1877642).

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 16:51:44 -0400

Changed in heat (Ubuntu Groovy):
status: Triaged → Fix Released

Reviewed: https://review.opendev.org/728104
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=4d5c7693cfd3f440733744032a60f00b5fa33cde
Submitter: Zuul
Branch: stable/ussuri

commit 4d5c7693cfd3f440733744032a60f00b5fa33cde
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:11:13 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I4929bfd4dc60f97b27459a4c6e7ed649c5e7f645
    Closes-Bug: #1863021

Reviewed: https://review.opendev.org/725346
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=68dc62b7389ca5896ca4bd5494b74f91e10b270f
Submitter: Zuul
Branch: master

commit 68dc62b7389ca5896ca4bd5494b74f91e10b270f
Author: Corey Bryant <email address hidden>
Date: Mon May 4 13:24:23 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I8c2ffea3c31b3009c87455d9c54dc233367f0d3d
    Closes-Bug: #1863021

Changed in designate:
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ironic - 1:15.0.0-0ubuntu0.20.04.1

---------------
ironic (1:15.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/725857. This fixes neutron service failures
    with Python 3.8 (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 13:06:46 -0400

Changed in ironic (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for ironic has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package heat - 1:14.0.0-0ubuntu0.20.04.1

---------------
heat (1:14.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 14.x series and get tarballs from opendev.org.
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/727181/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 16:51:44 -0400

Changed in heat (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package magnum - 10.0.0-0ubuntu0.20.04.1

---------------
magnum (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728010. This fixes service failures
    with Python 3.8 (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 12:31:17 -0400

Changed in magnum (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package designate - 1:10.0.0-0ubuntu0.20.04.1

---------------
designate (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Get tarballs from opendev.org and scope to 10.x series.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725346/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/p/disable-edgegrid.patch: Rebased.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Wed, 13 May 2020 16:50:55 -0400

Changed in designate (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ironic-inspector - 1:10.1.0-0ubuntu0.20.04.1

---------------
ironic-inspector (1:10.1.0-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/727184. This fixes service failures
    with Python 3.8 (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 09:57:10 +0000

Changed in ironic-inspector (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package barbican - 1:10.0.0-0ubuntu0.20.04.1

---------------
barbican (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * d/watch: Update to get release tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/watch: update to the correct version regex.
  * d/gbp.conf: Create stable/ussuri branch.

  [ Corey Bryant ]
  * d/watch: Scope to 10.x series.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725353/)
    to fix Python 3.8 monkey patching (LP: #1863021).

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 08:13:29 +0000

Changed in barbican (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openstack-trove - 1:13.0.0-0ubuntu0.20.04.1

---------------
openstack-trove (1:13.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/watch: Scope to 13.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725672/)
    to fix Python 3.8 monkey patching (LP: #1863021).

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 09:50:59 -0400

Changed in openstack-trove (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-mlnx - 1:15.0.2-0ubuntu0.20.04.1

---------------
networking-mlnx (1:15.0.2-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/control: Align (Build-)Depends with upstream.
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/725365)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 15:15:09 -0400

Changed in networking-mlnx (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-bagpipe - 12.0.0-0ubuntu0.20.04.1

---------------
networking-bagpipe (12.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 12.x series and get tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:52:15 -0400

Changed in networking-bagpipe (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-oslo.service - 2.1.1-0ubuntu1.1

---------------
python-oslo.service (2.1.1-0ubuntu1.1) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream review (https://review.opendev.org/#/c/725853/)
    to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 12:56:30 -0400

Changed in python-oslo.service (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-hyperv - 1:8.0.0-0ubuntu0.20.04.1

---------------
networking-hyperv (1:8.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:48:05 -0400

Changed in networking-hyperv (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-sfc - 10.0.0-0ubuntu0.20.04.1

---------------
networking-sfc (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  * d/watch: Scope to 10.x series and get tarballs from opendev.org.
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/control: Align (Build-)Depends with upstream.
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 14:41:04 -0400

Changed in networking-sfc (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mistral - 10.0.0-0ubuntu0.20.04.1

---------------
mistral (10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728369/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 10.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 09:37:52 -0400

Changed in mistral (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package senlin - 9.0.0-0ubuntu0.20.04.1

---------------
senlin (9.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/727186/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 9.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 10:02:46 -0400

Changed in senlin (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package masakari - 9.0.0-0ubuntu0.20.04.2

---------------
masakari (9.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * d/control, d/py3dist-overrides: Drop (Build-)Depends on
    universe package python3-sqlalchemy-utils. This was added in
    9.0.0 but only needed for upstream devstack gate job.

masakari (9.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728104/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 9.x series.
  * d/watch: Update to point at tarballs.opendev.org.
  * d/control: Align (Build-)Depends with upstream.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Tue, 19 May 2020 13:04:38 -0400

Changed in masakari (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-os-ken - 1.0.0-0ubuntu1.1

---------------
python-os-ken (1.0.0-0ubuntu1.1) focal; urgency=medium

  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked
    from upstream to fix Python 3.8 monkey patching (LP: #1863021).
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Thu, 14 May 2020 13:02:34 -0400

Changed in python-os-ken (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package manila - 1:10.0.0-0ubuntu0.20.04.1

---------------
manila (1:10.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728079/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 10.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/control: Align (Build-)Depends with upstream.
  * d/gbp.conf: Create stable/ussuri branch.

 -- Chris MacNaughton <email address hidden> Thu, 14 May 2020 13:36:55 +0000

Changed in manila (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package watcher - 1:4.0.0-0ubuntu0.20.04.1

---------------
watcher (1:4.0.0-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * New upstream release for OpenStack Ussuri (LP: #1877642).
  * d/p/monkey-patch-original-current-thread.patch: Cherry-picked from
    https://review.opendev.org/#/c/728397/. This fixes service failures
    with Python 3.8 (LP: #1863021)
  * d/watch: Scope to 4.x series.
  * d/watch: Update to point at tarballs.opendev.org.

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

 -- Corey Bryant <email address hidden> Fri, 15 May 2020 10:03:39 -0400

Changed in watcher (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in neutron (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in neutron-dynamic-routing (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in sahara (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in networking-l2gw (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in murano-agent (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in murano (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in cinder (Ubuntu Focal):
status: Fix Committed → Fix Released
tags: added: neutron-proactive-backport-potential
Dr. Jens Harbott (j-harbott) wrote :

Even with all these patches in, I'm still seeing messages like this with devstack running on focal, see https://review.opendev.org/704831:

Jun 01 16:23:06.645371 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[76054]: DEBUG neutron.agent.linux.utils [-] Running command: ['env', 'LC_ALL=C', 'PATH=/sbin:/us
r/sbin', 'dnsmasq', '--test', '--dhcp-host=tag:foo'] {{(pid=76054) create_process /opt/stack/neutron/neutron/agent/linux/utils.py:87}}
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: Exception ignored in: <function _after_fork at 0x7fc960125d30>
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: Traceback (most recent call last):
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: assert len(_active) == 1
Jun 01 16:23:06.651565 ubuntu-focal-rax-dfw-0016894858 neutron-dhcp-agent[78057]: AssertionError:

This happens for neutron-dhcp-agent, neutron-ovs-agent, nova-compute and cinder-volume, seemingly always on a call of subprocess.Popen.

sean mooney (sean-k-mooney) wrote :

im not sure that https://review.opendev.org/#/c/707474/ acttully works
or at the very least is a complete fix.

as noted in https://bugs.launchpad.net/nova/+bug/1882521/comments/1
we still see the same threading error.

i think we likely need to patch oslo_concurrency too or look into another fix.

Change abandoned by Qiu Fossen (<email address hidden>) on branch: master
Review: https://review.opendev.org/726333

Hmmm .. not sure whether this patch fixes all scenarios with nova-compute:

We just created a new compute server from scratch with clean install ubuntu focal, python 3.8 and Ussuri nova-compute (nova-compute/focal-updates,now 2:21.0.0-0ubuntu0.20.04.1).
I double checked that this patch has been applied (which is in /usr/lib/python3/dist-upgrade/nova/monkey_patch.py).
However, my syslog is spammed with
Jun 14 09:53:20 compute3 nova-compute[2759819]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759824]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759824]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759824]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759824]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759824]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759827]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759827]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759827]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759827]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759827]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759830]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759830]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759830]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759830]: assert len(_active) == 1
Jun 14 09:53:20 compute3 nova-compute[2759830]: AssertionError:
Jun 14 09:53:20 compute3 nova-compute[2759833]: Exception ignored in: <function _after_fork at 0x7f91837ae820>
Jun 14 09:53:20 compute3 nova-compute[2759833]: Traceback (most recent call last):
Jun 14 09:53:20 compute3 nova-compute[2759833]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 14 09:53:20 compute3 nova-compute[2759833]: assert len(_active) == 1
nova-compute ist running and looked to be functional.

Did I overlook something?
Thanks for your reply

You may be right.

I have this in monkey_patch.py:

    # Monkey patch the original current_thread to use the up-to-date _active
    # global variable. See https://bugs.launchpad.net/bugs/1863021 and
    # https://github.com/eventlet/eventlet/issues/592
    import __original_module_threading as orig_threading
    import threading
    orig_threading.current_thread.__globals__['_active'] = threading._active

However, nova-compute logs still get spammed with:

Jun 16 14:06:30 node nova-compute[12592]: Traceback (most recent call last):
Jun 16 14:06:30 node nova-compute[12592]: File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Jun 16 14:06:30 node nova-compute[12592]: assert len(_active) == 1
Jun 16 14:06:30 node nova-compute[12592]: AssertionError:
Jun 16 14:06:30 node nova-compute[12592]: Exception ignored in: <function _after_fork at 0x7efdc80b7b80>

Tim Burke (1-tim-z) wrote :

I wonder if an approach like in my https://github.com/eventlet/eventlet/pull/611 might work better.

Reviewed: https://review.opendev.org/725672
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=56bb6f39b6c808160d2e4c96fcc2104ba1daf817
Submitter: Zuul
Branch: master

commit 56bb6f39b6c808160d2e4c96fcc2104ba1daf817
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:42:31 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I61e5e270bf66b0355da3282c19cbc9fd42c4090b
    Closes-Bug: #1863021

Changed in trove:
status: In Progress → Fix Released
tags: removed: neutron-proactive-backport-potential

Reviewed: https://review.opendev.org/727186
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=fb738fcd276e3d0aeee975964aaa576c51c0842e
Submitter: Zuul
Branch: stable/ussuri

commit fb738fcd276e3d0aeee975964aaa576c51c0842e
Author: Corey Bryant <email address hidden>
Date: Tue May 5 15:46:38 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I5c2fbe1827195d670a67dd62cdadce5ee3513ace
    Closes-Bug: #1863021
    (cherry picked from commit 666fd70b6100e329e7eaec82e808ad0722d67048)

Reviewed: https://review.opendev.org/743431
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=779ae5f1f1cd8328d1423f13550c21fe53b0c803
Submitter: Zuul
Branch: stable/train

commit 779ae5f1f1cd8328d1423f13550c21fe53b0c803
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:04:40 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: Ifc6420d927c0ce9e04ff3b3253e81a474591e9bb
    Closes-Bug: #1863021
    (cherry picked from commit 5e9f694a5a8f90c72680acb1181318930f55aa30)
    (cherry picked from commit 7196cfcb091d66856a1ca0db53816198249196db)

Reviewed: https://review.opendev.org/726073
Committed: https://git.openstack.org/cgit/openstack/networking-bagpipe/commit/?id=0e866eed9f6c3e545a2fd6c0b680a718996661a5
Submitter: Zuul
Branch: stable/ussuri

commit 0e866eed9f6c3e545a2fd6c0b680a718996661a5
Author: Corey Bryant <email address hidden>
Date: Mon May 4 14:18:49 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I5ea4bb27361df3c489deedb51d7ca8ea64bb923b
    Closes-Bug: #1863021
    (cherry picked from commit f0db421843f1161f4daa1e08b83eb2b11e78d9dd)

Reviewed: https://review.opendev.org/725790
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=56de1c4330de76256937ecd0b1d8cde8f16d9f0a
Submitter: Zuul
Branch: stable/train

commit 56de1c4330de76256937ecd0b1d8cde8f16d9f0a
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:31:09 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I8e8a9ce5085b7915994317af5ea903cf8a39e809
    Closes-Bug: #1863021
    (cherry picked from commit 44e77ea9a975e9222f38591151e2c32640ef7379)

Reviewed: https://review.opendev.org/725397
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=97e658417b691f6f737599c59726cc49131b05ef
Submitter: Zuul
Branch: master

commit 97e658417b691f6f737599c59726cc49131b05ef
Author: Corey Bryant <email address hidden>
Date: Mon May 4 17:11:13 2020 -0400

    Monkey patch original current_thread _active

    Monkey patch the original current_thread to use the up-to-date _active
    global variable. This solution is based on that documented at:
    https://github.com/eventlet/eventlet/issues/592

    Change-Id: I4929bfd4dc60f97b27459a4c6e7ed649c5e7f645
    Closes-Bug: #1863021

Changed in masakari:
status: In Progress → Fix Released
Displaying first 40 and last 40 comments. View all 142 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.