[SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
BaGPipe |
Fix Released
|
Undecided
|
Unassigned | ||
Cinder |
Fix Released
|
Undecided
|
Corey Bryant | ||
Designate |
Fix Released
|
Undecided
|
Corey Bryant | ||
Glance |
Fix Released
|
Undecided
|
Unassigned | ||
Mellanox backend integration with Neutron (networking-mlnx) |
Fix Released
|
Undecided
|
Unassigned | ||
Mistral |
Fix Released
|
Undecided
|
Unassigned | ||
Murano |
Fix Released
|
Undecided
|
Corey Bryant | ||
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Corey Bryant | ||
OpenStack DBaaS (Trove) |
Fix Released
|
Undecided
|
Lingxian Kong | ||
OpenStack Object Storage (swift) |
In Progress
|
Undecided
|
Chris MacNaughton | ||
OpenStack Shared File Systems Service (Manila) |
Fix Released
|
Undecided
|
suzhengwei | ||
masakari |
Fix Released
|
Undecided
|
Corey Bryant | ||
networking-hyperv |
Fix Released
|
Undecided
|
Corey Bryant | ||
networking-l2gw |
Fix Released
|
Undecided
|
Unassigned | ||
networking-sfc |
Fix Released
|
Undecided
|
Unassigned | ||
neutron |
Fix Released
|
High
|
Dr. Jens Harbott | ||
oslo.service |
Fix Released
|
Undecided
|
Corey Bryant | ||
senlin |
Fix Released
|
Undecided
|
Unassigned | ||
watcher |
Fix Released
|
Undecided
|
Chris MacNaughton | ||
barbican (Ubuntu) |
Fix Released
|
High
|
Arthur Dayne | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Arthur Dayne | ||
cinder (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
designate (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
glance (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
heat (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
ironic (Ubuntu) |
Fix Released
|
High
|
Arthur Dayne | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Arthur Dayne | ||
ironic-inspector (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
magnum (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
manila (Ubuntu) |
Fix Released
|
High
|
Arthur Dayne | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Arthur Dayne | ||
masakari (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
mistral (Ubuntu) |
Fix Released
|
High
|
Arthur Dayne | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Arthur Dayne | ||
murano (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
murano-agent (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
networking-bagpipe (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
networking-hyperv (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
networking-l2gw (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
networking-mlnx (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
networking-sfc (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
neutron (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
neutron-dynamic-routing (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
nova (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
openstack-trove (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
python-os-ken (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
python-oslo.service (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
sahara (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
senlin (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned | ||
swift (Ubuntu) |
Triaged
|
Low
|
Unassigned | ||
Focal |
Triaged
|
Low
|
Unassigned | ||
watcher (Ubuntu) |
Fix Released
|
High
|
Arthur Dayne | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Arthur Dayne |
Bug Description
[Impact]
This appears to be the same issue documented here: https:/
However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:
Exception ignored in: <function _after_fork at 0x7f1100d075e0>
Traceback (most recent call last):
File "/usr/lib/
assert len(_active) == 1
AssertionError:
Exception ignored in: <function _after_fork at 0x7f1100d075e0>
Traceback (most recent call last):
File "/usr/lib/
assert len(_active) == 1
AssertionError:
Patching nova/monkey_
diff --git a/nova/
index a07ff91dac.
--- a/nova/
+++ b/nova/
@@ -59,6 +59,9 @@ def _monkey_patch():
else:
+ import __original_
+ import threading
+ __original_
# NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
# hub use a monotonic clock to avoid issues with drifts of system time (see
Similar patches to glance/cmd/api.py, glance/
[Test Case]
Tempest regression on a deployed openstack should pass successfully.
Individual services can be tested as follows but with the scope of this issue tempest regression testing will be the goal for verifying this SRU.
'systemctl status <service>' should not show the failure reported in
[Impact].
[Regression Potential]
Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.
Changed in glance (Ubuntu): | |
status: | New → Triaged |
importance: | Undecided → High |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in cinder (Ubuntu): | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in neutron (Ubuntu): | |
importance: | Undecided → High |
status: | New → Triaged |
summary: |
- eventlet monkey patch results in assert len(_active) == 1 AssertionError + [SRU] eventlet monkey patch results in assert len(_active) == 1 + AssertionError |
description: | updated |
Changed in neutron (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in cinder (Ubuntu Focal): | |
status: | New → Fix Released |
importance: | Undecided → High |
status: | Fix Released → Triaged |
Changed in glance (Ubuntu Focal): | |
status: | New → Fix Released |
Changed in nova (Ubuntu Focal): | |
status: | New → Fix Released |
Changed in neutron: | |
importance: | Undecided → High |
Changed in heat (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in heat (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in swift (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in swift (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in designate (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in designate (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in barbican (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in barbican (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in ironic (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in ironic (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in ironic-inspector (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in ironic-inspector (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in python-oslo.service (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in python-oslo.service (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-bagpipe (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-bagpipe (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-l2gw (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-l2gw (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-hyperv (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-hyperv (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-mlnx (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-mlnx (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-sfc (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in magnum (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in magnum (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in manila (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in manila (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in masakari (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in masakari (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in mistral (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in mistral (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in neutron-dynamic-routing (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in neutron-dynamic-routing (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in neutron: | |
assignee: | Corey Bryant (corey.bryant) → Dr. Jens Harbott (j-harbott) |
Changed in murano (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in murano (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in murano-agent (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in murano-agent (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in openstack-trove (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in openstack-trove (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in sahara (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in sahara (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in senlin (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in senlin (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in networking-sfc (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in swift: | |
assignee: | nobody → Chris MacNaughton (chris.macnaughton) |
Changed in watcher: | |
assignee: | nobody → Chris MacNaughton (chris.macnaughton) |
status: | New → In Progress |
Changed in networking-l2gw: | |
status: | In Progress → Fix Released |
Changed in networking-bagpipe: | |
status: | In Progress → Fix Released |
Changed in networking-sfc: | |
status: | In Progress → Fix Released |
Changed in networking-hyperv: | |
status: | In Progress → Fix Released |
Changed in python-os-ken (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in python-os-ken (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in watcher (Ubuntu Focal): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in watcher (Ubuntu Groovy): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in barbican (Ubuntu Groovy): | |
assignee: | nobody → Arthur Dayne (palagend) |
Changed in mistral (Ubuntu Groovy): | |
assignee: | nobody → Arthur Dayne (palagend) |
Changed in ironic (Ubuntu Groovy): | |
assignee: | nobody → Arthur Dayne (palagend) |
Changed in manila (Ubuntu Groovy): | |
assignee: | nobody → Arthur Dayne (palagend) |
Changed in watcher (Ubuntu Groovy): | |
assignee: | nobody → Arthur Dayne (palagend) |
Changed in manila: | |
assignee: | Corey Bryant (corey.bryant) → suzhengwei (sue.sam) |
Changed in senlin: | |
status: | In Progress → Fix Released |
Changed in swift (Ubuntu Focal): | |
importance: | High → Low |
Changed in swift (Ubuntu Groovy): | |
importance: | High → Low |
Changed in ironic (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in mistral: | |
status: | In Progress → Fix Released |
Changed in watcher: | |
status: | In Progress → Fix Released |
Changed in designate (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
description: | updated |
Changed in barbican (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in networking-hyperv (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in networking-bagpipe (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in ironic-inspector (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in manila (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in python-os-ken (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in networking-sfc (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in python-oslo.service (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in networking-mlnx (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in mistral (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in senlin (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in watcher (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in openstack-trove (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in murano (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in murano-agent (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in networking-l2gw (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in neutron-dynamic-routing (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in ironic (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
tags: | added: verification-needed verification-needed-focal |
Changed in magnum (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in manila: | |
status: | In Progress → Fix Released |
Changed in heat (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in designate (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in barbican (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in ironic-inspector (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in magnum (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in openstack-trove (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in masakari (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in networking-mlnx (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in networking-bagpipe (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in python-os-ken (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in python-oslo.service (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in networking-hyperv (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in networking-sfc (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in manila (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in mistral (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in senlin (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in masakari (Ubuntu Focal): | |
status: | Fix Committed → Triaged |
Changed in watcher (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in masakari (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in masakari (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in trove: | |
assignee: | Corey Bryant (corey.bryant) → Lingxian Kong (kong) |
tags: |
added: verification-done verification-done-focal removed: verification-needed verification-needed-focal |
Changed in cinder (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in murano (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in murano-agent (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in networking-l2gw (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in neutron (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in neutron-dynamic-routing (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in sahara (Ubuntu Focal): | |
status: | Triaged → Fix Committed |
Changed in sahara (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in heat (Ubuntu Groovy): | |
status: | Triaged → Fix Released |
Changed in designate: | |
status: | In Progress → Fix Released |
Changed in ironic (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in neutron (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in neutron-dynamic-routing (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in sahara (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in networking-l2gw (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in murano-agent (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in murano (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
Changed in cinder (Ubuntu Focal): | |
status: | Fix Committed → Fix Released |
tags: | added: neutron-proactive-backport-potential |
tags: | removed: neutron-proactive-backport-potential |
no longer affects: | swift (Ubuntu Groovy) |
See attached for much more detailed recreation/ patching/ fixing.