Activity log for bug #1863021

Date Who What changed Old value New value Message
2020-02-12 21:14:03 Corey Bryant bug added bug
2020-02-12 21:16:54 Corey Bryant tags py38
2020-02-12 21:18:12 Corey Bryant attachment added nova-eventlet-monkey-patch.txt https://bugs.launchpad.net/nova/+bug/1863021/+attachment/5327763/+files/nova-eventlet-monkey-patch.txt
2020-02-12 21:18:30 Corey Bryant bug task added nova (Ubuntu)
2020-02-12 21:18:37 Corey Bryant nova (Ubuntu): status New Triaged
2020-02-12 21:18:39 Corey Bryant nova (Ubuntu): importance Undecided High
2020-02-12 21:35:08 OpenStack Infra nova: status New In Progress
2020-02-12 21:35:08 OpenStack Infra nova: assignee Corey Bryant (corey.bryant)
2020-02-14 11:08:50 Dr. Jens Harbott bug added subscriber Dr. Jens Harbott
2020-02-21 02:17:47 Launchpad Janitor nova (Ubuntu): status Triaged Fix Released
2020-03-03 02:19:31 OpenStack Infra nova: status In Progress Fix Released
2020-03-03 02:19:34 OpenStack Infra bug watch added https://github.com/eventlet/eventlet/issues/592
2020-03-30 18:49:06 Corey Bryant bug task added glance (Ubuntu)
2020-03-30 18:49:22 Corey Bryant bug task added glance
2020-03-30 18:49:32 Corey Bryant glance (Ubuntu): status New Triaged
2020-03-30 18:49:36 Corey Bryant glance (Ubuntu): importance Undecided High
2020-03-30 18:50:05 Corey Bryant description This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova services fail with: Exception ignored in: <function _after_fork at 0x7f1100d075e0> Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork assert len(_active) == 1 AssertionError: Exception ignored in: <function _after_fork at 0x7f1100d075e0> Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork assert len(_active) == 1 AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch(): else: eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet # hub use a monotonic clock to avoid issues with drifts of system time (see This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see
2020-03-30 19:01:21 Corey Bryant description This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patch to glance/cmd/api.py appears to fix it for glance.
2020-03-30 19:11:17 Corey Bryant description This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patch to glance/cmd/api.py appears to fix it for glance. This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance.
2020-03-30 23:52:22 Launchpad Janitor glance (Ubuntu): status Triaged Fix Released
2020-03-31 12:53:54 OpenStack Infra glance: status In Progress Fix Released
2020-04-30 19:13:47 Corey Bryant bug task added cinder
2020-04-30 19:14:02 Corey Bryant bug task added cinder (Ubuntu)
2020-04-30 19:15:52 Corey Bryant description This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova and glance services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance and cinder services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance.
2020-04-30 19:16:02 Corey Bryant cinder (Ubuntu): status New Triaged
2020-04-30 19:16:07 Corey Bryant cinder (Ubuntu): importance Undecided High
2020-04-30 19:49:20 Corey Bryant bug task added neutron
2020-04-30 20:40:44 OpenStack Infra neutron: status New In Progress
2020-04-30 20:40:44 OpenStack Infra neutron: assignee Corey Bryant (corey.bryant)
2020-04-30 20:41:01 OpenStack Infra cinder: status New In Progress
2020-04-30 20:41:01 OpenStack Infra cinder: assignee Corey Bryant (corey.bryant)
2020-05-01 14:01:49 Corey Bryant bug task added neutron (Ubuntu)
2020-05-01 14:02:09 Corey Bryant neutron (Ubuntu): importance Undecided High
2020-05-01 14:02:09 Corey Bryant neutron (Ubuntu): status New Triaged
2020-05-01 16:07:50 Corey Bryant summary eventlet monkey patch results in assert len(_active) == 1 AssertionError [SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError
2020-05-01 16:11:55 Corey Bryant description This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance and cinder services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. [Impact] This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. [Test Case] 'systemctl status <service>' should not show the failure reported in [Impact]. [Regression Potential] Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.
2020-05-01 16:19:15 Corey Bryant nominated for series Ubuntu Groovy
2020-05-01 16:19:15 Corey Bryant bug task added nova (Ubuntu Groovy)
2020-05-01 16:19:15 Corey Bryant bug task added glance (Ubuntu Groovy)
2020-05-01 16:19:15 Corey Bryant bug task added cinder (Ubuntu Groovy)
2020-05-01 16:19:15 Corey Bryant bug task added neutron (Ubuntu Groovy)
2020-05-01 16:19:15 Corey Bryant nominated for series Ubuntu Focal
2020-05-01 16:19:15 Corey Bryant bug task added nova (Ubuntu Focal)
2020-05-01 16:19:15 Corey Bryant bug task added glance (Ubuntu Focal)
2020-05-01 16:19:15 Corey Bryant bug task added cinder (Ubuntu Focal)
2020-05-01 16:19:15 Corey Bryant bug task added neutron (Ubuntu Focal)
2020-05-01 16:19:45 Corey Bryant neutron (Ubuntu Focal): importance Undecided High
2020-05-01 16:19:45 Corey Bryant neutron (Ubuntu Focal): status New Triaged
2020-05-01 16:20:17 Corey Bryant cinder (Ubuntu Focal): status New Fix Released
2020-05-01 16:20:33 Corey Bryant cinder (Ubuntu Focal): importance Undecided High
2020-05-01 16:20:53 Corey Bryant cinder (Ubuntu Focal): status Fix Released Triaged
2020-05-01 16:21:29 Corey Bryant glance (Ubuntu Focal): status New Fix Released
2020-05-01 16:21:47 Corey Bryant nova (Ubuntu Focal): status New Fix Released
2020-05-01 20:40:29 Corey Bryant bug task added swift
2020-05-01 20:45:04 Corey Bryant bug task added swift (Ubuntu)
2020-05-01 20:45:16 Corey Bryant bug task added heat (Ubuntu)
2020-05-02 04:49:00 Launchpad Janitor neutron (Ubuntu Groovy): status Triaged Fix Released
2020-05-04 08:59:16 Akihiro Motoki neutron: importance Undecided High
2020-05-04 14:09:22 Corey Bryant heat (Ubuntu Focal): importance Undecided High
2020-05-04 14:09:22 Corey Bryant heat (Ubuntu Focal): status New Triaged
2020-05-04 14:09:39 Corey Bryant heat (Ubuntu Groovy): importance Undecided High
2020-05-04 14:09:39 Corey Bryant heat (Ubuntu Groovy): status New Triaged
2020-05-04 14:10:33 Corey Bryant swift (Ubuntu Focal): importance Undecided High
2020-05-04 14:10:33 Corey Bryant swift (Ubuntu Focal): status New Triaged
2020-05-04 14:10:52 Corey Bryant swift (Ubuntu Groovy): importance Undecided High
2020-05-04 14:10:52 Corey Bryant swift (Ubuntu Groovy): status New Triaged
2020-05-04 15:56:56 Launchpad Janitor cinder (Ubuntu Groovy): status Triaged Fix Released
2020-05-04 17:23:05 Corey Bryant bug task added designate
2020-05-04 17:23:28 Corey Bryant bug task added designate (Ubuntu)
2020-05-04 17:23:44 Corey Bryant designate (Ubuntu Focal): importance Undecided High
2020-05-04 17:23:44 Corey Bryant designate (Ubuntu Focal): status New Triaged
2020-05-04 17:24:02 Corey Bryant designate (Ubuntu Groovy): importance Undecided High
2020-05-04 17:24:02 Corey Bryant designate (Ubuntu Groovy): status New Triaged
2020-05-04 17:24:57 OpenStack Infra designate: status New In Progress
2020-05-04 17:24:57 OpenStack Infra designate: assignee Corey Bryant (corey.bryant)
2020-05-04 17:36:17 Corey Bryant bug task added barbican (Ubuntu)
2020-05-04 17:36:34 Corey Bryant barbican (Ubuntu Focal): importance Undecided High
2020-05-04 17:36:34 Corey Bryant barbican (Ubuntu Focal): status New Triaged
2020-05-04 17:36:53 Corey Bryant barbican (Ubuntu Groovy): importance Undecided High
2020-05-04 17:36:53 Corey Bryant barbican (Ubuntu Groovy): status New Triaged
2020-05-04 17:42:54 Corey Bryant bug task added ironic (Ubuntu)
2020-05-04 17:43:13 Corey Bryant ironic (Ubuntu Focal): importance Undecided High
2020-05-04 17:43:13 Corey Bryant ironic (Ubuntu Focal): status New Triaged
2020-05-04 17:43:32 Corey Bryant ironic (Ubuntu Groovy): importance Undecided High
2020-05-04 17:43:32 Corey Bryant ironic (Ubuntu Groovy): status New Triaged
2020-05-04 17:50:11 Corey Bryant bug task added ironic-inspector (Ubuntu)
2020-05-04 17:50:31 Corey Bryant ironic-inspector (Ubuntu Focal): importance Undecided High
2020-05-04 17:50:31 Corey Bryant ironic-inspector (Ubuntu Focal): status New Triaged
2020-05-04 17:50:49 Corey Bryant ironic-inspector (Ubuntu Groovy): importance Undecided High
2020-05-04 17:50:49 Corey Bryant ironic-inspector (Ubuntu Groovy): status New Triaged
2020-05-04 18:05:08 Corey Bryant bug task added oslo.service
2020-05-04 18:05:30 Corey Bryant bug task added python-oslo.service (Ubuntu)
2020-05-04 18:05:51 Corey Bryant python-oslo.service (Ubuntu Focal): importance Undecided High
2020-05-04 18:05:51 Corey Bryant python-oslo.service (Ubuntu Focal): status New Triaged
2020-05-04 18:06:11 Corey Bryant python-oslo.service (Ubuntu Groovy): importance Undecided High
2020-05-04 18:06:11 Corey Bryant python-oslo.service (Ubuntu Groovy): status New Triaged
2020-05-04 18:07:40 OpenStack Infra oslo.service: status New In Progress
2020-05-04 18:07:40 OpenStack Infra oslo.service: assignee Corey Bryant (corey.bryant)
2020-05-04 18:17:25 Corey Bryant bug task added networking-bagpipe
2020-05-04 18:17:45 Corey Bryant bug task added networking-bagpipe (Ubuntu)
2020-05-04 18:18:10 Corey Bryant networking-bagpipe (Ubuntu Focal): importance Undecided High
2020-05-04 18:18:10 Corey Bryant networking-bagpipe (Ubuntu Focal): status New Triaged
2020-05-04 18:18:28 Corey Bryant networking-bagpipe (Ubuntu Groovy): importance Undecided High
2020-05-04 18:18:28 Corey Bryant networking-bagpipe (Ubuntu Groovy): status New Triaged
2020-05-04 18:20:35 Corey Bryant bug task added networking-hyperv
2020-05-04 18:21:28 OpenStack Infra networking-hyperv: status New In Progress
2020-05-04 18:21:28 OpenStack Infra networking-hyperv: assignee Corey Bryant (corey.bryant)
2020-05-04 18:22:33 Corey Bryant bug task added networking-hyperv (Ubuntu)
2020-05-04 18:22:56 Corey Bryant bug task added networking-l2gw
2020-05-04 18:24:12 Corey Bryant bug task added networking-l2gw (Ubuntu)
2020-05-04 18:24:40 Corey Bryant networking-l2gw (Ubuntu Focal): importance Undecided High
2020-05-04 18:24:40 Corey Bryant networking-l2gw (Ubuntu Focal): status New Triaged
2020-05-04 18:25:09 Corey Bryant networking-l2gw (Ubuntu Groovy): importance Undecided High
2020-05-04 18:25:09 Corey Bryant networking-l2gw (Ubuntu Groovy): status New Triaged
2020-05-04 18:25:32 Corey Bryant networking-hyperv (Ubuntu Focal): importance Undecided High
2020-05-04 18:25:32 Corey Bryant networking-hyperv (Ubuntu Focal): status New Triaged
2020-05-04 18:26:29 Corey Bryant networking-hyperv (Ubuntu Groovy): importance Undecided High
2020-05-04 18:26:29 Corey Bryant networking-hyperv (Ubuntu Groovy): status New Triaged
2020-05-04 18:28:16 Corey Bryant bug task added networking-mlnx
2020-05-04 18:28:42 Corey Bryant bug task added networking-mlnx (Ubuntu)
2020-05-04 18:29:14 Corey Bryant networking-mlnx (Ubuntu Focal): importance Undecided High
2020-05-04 18:29:14 Corey Bryant networking-mlnx (Ubuntu Focal): status New Triaged
2020-05-04 18:29:43 Corey Bryant networking-mlnx (Ubuntu Groovy): importance Undecided High
2020-05-04 18:29:43 Corey Bryant networking-mlnx (Ubuntu Groovy): status New Triaged
2020-05-04 18:31:10 Corey Bryant bug task added networking-sfc
2020-05-04 18:32:29 Corey Bryant bug task added networking-sfc (Ubuntu)
2020-05-04 18:32:53 Corey Bryant networking-sfc (Ubuntu Focal): importance Undecided High
2020-05-04 18:32:53 Corey Bryant networking-sfc (Ubuntu Focal): status New Triaged
2020-05-04 20:56:39 Corey Bryant bug task added magnum (Ubuntu)
2020-05-04 20:56:58 Corey Bryant magnum (Ubuntu Focal): importance Undecided High
2020-05-04 20:56:58 Corey Bryant magnum (Ubuntu Focal): status New Triaged
2020-05-04 20:57:53 Corey Bryant magnum (Ubuntu Groovy): importance Undecided High
2020-05-04 20:57:53 Corey Bryant magnum (Ubuntu Groovy): status New Triaged
2020-05-04 21:05:11 Corey Bryant bug task added manila
2020-05-04 21:06:23 Corey Bryant bug task added manila (Ubuntu)
2020-05-04 21:06:47 Corey Bryant manila (Ubuntu Focal): importance Undecided High
2020-05-04 21:06:47 Corey Bryant manila (Ubuntu Focal): status New Triaged
2020-05-04 21:07:21 OpenStack Infra manila: status New In Progress
2020-05-04 21:07:21 OpenStack Infra manila: assignee Corey Bryant (corey.bryant)
2020-05-04 21:07:31 Corey Bryant manila (Ubuntu Groovy): importance Undecided High
2020-05-04 21:07:31 Corey Bryant manila (Ubuntu Groovy): status New Triaged
2020-05-04 21:10:43 Corey Bryant bug task added masakari
2020-05-04 21:11:00 Corey Bryant bug task added masakari (Ubuntu)
2020-05-04 21:11:35 Corey Bryant masakari (Ubuntu Focal): importance Undecided High
2020-05-04 21:11:35 Corey Bryant masakari (Ubuntu Focal): status New Triaged
2020-05-04 21:11:42 OpenStack Infra masakari: status New In Progress
2020-05-04 21:11:42 OpenStack Infra masakari: assignee Corey Bryant (corey.bryant)
2020-05-04 21:11:58 Corey Bryant masakari (Ubuntu Groovy): importance Undecided High
2020-05-04 21:11:58 Corey Bryant masakari (Ubuntu Groovy): status New Triaged
2020-05-04 21:15:59 Corey Bryant bug task added mistral
2020-05-04 21:16:45 Corey Bryant bug task added mistral (Ubuntu)
2020-05-04 21:17:08 Corey Bryant mistral (Ubuntu Focal): importance Undecided High
2020-05-04 21:17:08 Corey Bryant mistral (Ubuntu Focal): status New Triaged
2020-05-04 21:17:32 Corey Bryant mistral (Ubuntu Groovy): importance Undecided High
2020-05-04 21:17:32 Corey Bryant mistral (Ubuntu Groovy): status New Triaged
2020-05-04 21:29:57 Corey Bryant bug task added neutron-dynamic-routing (Ubuntu)
2020-05-04 21:30:22 Corey Bryant neutron-dynamic-routing (Ubuntu Focal): importance Undecided High
2020-05-04 21:30:22 Corey Bryant neutron-dynamic-routing (Ubuntu Focal): status New Triaged
2020-05-04 21:30:49 Corey Bryant neutron-dynamic-routing (Ubuntu Groovy): importance Undecided High
2020-05-04 21:30:49 Corey Bryant neutron-dynamic-routing (Ubuntu Groovy): status New Triaged
2020-05-05 06:19:30 OpenStack Infra neutron: assignee Corey Bryant (corey.bryant) Dr. Jens Harbott (j-harbott)
2020-05-05 14:19:23 OpenStack Infra oslo.service: status In Progress Fix Released
2020-05-05 15:33:29 OpenStack Infra neutron: status In Progress Fix Released
2020-05-05 19:26:44 Corey Bryant bug task added murano
2020-05-05 19:27:46 Corey Bryant bug task added murano (Ubuntu)
2020-05-05 19:28:04 Corey Bryant bug task added murano-agent (Ubuntu)
2020-05-05 19:28:31 Corey Bryant bug task added trove
2020-05-05 19:29:02 Corey Bryant bug task added senlin
2020-05-05 19:29:46 Corey Bryant bug task added senlin (Ubuntu)
2020-05-05 19:30:06 Corey Bryant bug task added sahara (Ubuntu)
2020-05-05 19:31:03 Corey Bryant bug task added openstack-trove (Ubuntu)
2020-05-05 19:32:46 Corey Bryant murano (Ubuntu Focal): importance Undecided High
2020-05-05 19:32:46 Corey Bryant murano (Ubuntu Focal): status New Triaged
2020-05-05 19:33:15 Corey Bryant murano (Ubuntu Groovy): importance Undecided High
2020-05-05 19:33:15 Corey Bryant murano (Ubuntu Groovy): status New Triaged
2020-05-05 19:33:39 Corey Bryant murano-agent (Ubuntu Focal): importance Undecided High
2020-05-05 19:33:39 Corey Bryant murano-agent (Ubuntu Focal): status New Triaged
2020-05-05 19:34:04 Corey Bryant murano-agent (Ubuntu Groovy): importance Undecided High
2020-05-05 19:34:04 Corey Bryant murano-agent (Ubuntu Groovy): status New Triaged
2020-05-05 19:34:27 Corey Bryant openstack-trove (Ubuntu Focal): importance Undecided High
2020-05-05 19:34:27 Corey Bryant openstack-trove (Ubuntu Focal): status New Triaged
2020-05-05 19:34:49 Corey Bryant openstack-trove (Ubuntu Groovy): importance Undecided High
2020-05-05 19:34:49 Corey Bryant openstack-trove (Ubuntu Groovy): status New Triaged
2020-05-05 19:35:12 Corey Bryant sahara (Ubuntu Focal): importance Undecided High
2020-05-05 19:35:12 Corey Bryant sahara (Ubuntu Focal): status New Triaged
2020-05-05 19:35:38 Corey Bryant sahara (Ubuntu Groovy): importance Undecided High
2020-05-05 19:35:38 Corey Bryant sahara (Ubuntu Groovy): status New Triaged
2020-05-05 19:36:02 Corey Bryant senlin (Ubuntu Focal): importance Undecided High
2020-05-05 19:36:02 Corey Bryant senlin (Ubuntu Focal): status New Triaged
2020-05-05 19:36:28 Corey Bryant senlin (Ubuntu Groovy): importance Undecided High
2020-05-05 19:36:28 Corey Bryant senlin (Ubuntu Groovy): status New Triaged
2020-05-05 19:36:55 Corey Bryant networking-sfc (Ubuntu Groovy): importance Undecided High
2020-05-05 19:36:55 Corey Bryant networking-sfc (Ubuntu Groovy): status New Triaged
2020-05-05 19:40:36 OpenStack Infra murano: status New In Progress
2020-05-05 19:40:36 OpenStack Infra murano: assignee Corey Bryant (corey.bryant)
2020-05-05 19:42:56 OpenStack Infra trove: status New In Progress
2020-05-05 19:42:56 OpenStack Infra trove: assignee Corey Bryant (corey.bryant)
2020-05-05 21:01:35 OpenStack Infra cinder: status In Progress Fix Released
2020-05-05 21:29:47 OpenStack Infra murano: status In Progress Fix Released
2020-05-06 06:23:01 Chris MacNaughton swift: assignee Chris MacNaughton (chris.macnaughton)
2020-05-06 06:25:25 Chris MacNaughton bug task added watcher
2020-05-06 06:25:45 Chris MacNaughton bug task added watcher (Ubuntu)
2020-05-06 06:26:00 Chris MacNaughton watcher: status New In Progress
2020-05-06 06:26:00 Chris MacNaughton watcher: assignee Chris MacNaughton (chris.macnaughton)
2020-05-06 10:01:24 OpenStack Infra tags py38 in-stable-ussuri py38
2020-05-06 13:10:51 Chris MacNaughton bug task added python-os-ken (Ubuntu)
2020-05-06 23:11:57 OpenStack Infra tags in-stable-ussuri py38 in-stable-train in-stable-ussuri py38
2020-05-07 09:05:08 OpenStack Infra networking-l2gw: status In Progress Fix Released
2020-05-07 09:19:22 OpenStack Infra networking-bagpipe: status In Progress Fix Released
2020-05-07 09:56:06 OpenStack Infra networking-sfc: status In Progress Fix Released
2020-05-07 13:00:40 OpenStack Infra networking-hyperv: status In Progress Fix Released
2020-05-07 15:18:40 Corey Bryant python-os-ken (Ubuntu Focal): importance Undecided High
2020-05-07 15:18:40 Corey Bryant python-os-ken (Ubuntu Focal): status New Triaged
2020-05-07 15:19:39 Corey Bryant python-os-ken (Ubuntu Groovy): importance Undecided High
2020-05-07 15:19:39 Corey Bryant python-os-ken (Ubuntu Groovy): status New Triaged
2020-05-07 15:20:05 Corey Bryant watcher (Ubuntu Focal): importance Undecided High
2020-05-07 15:20:05 Corey Bryant watcher (Ubuntu Focal): status New Triaged
2020-05-07 15:20:51 Corey Bryant watcher (Ubuntu Groovy): importance Undecided High
2020-05-07 15:20:51 Corey Bryant watcher (Ubuntu Groovy): status New Triaged
2020-05-08 09:38:16 Arthur Dayne barbican (Ubuntu Groovy): assignee Arthur Dayne (palagend)
2020-05-08 09:39:29 Arthur Dayne mistral (Ubuntu Groovy): assignee Arthur Dayne (palagend)
2020-05-08 09:41:11 Arthur Dayne ironic (Ubuntu Groovy): assignee Arthur Dayne (palagend)
2020-05-08 10:30:04 Arthur Dayne manila (Ubuntu Groovy): assignee Arthur Dayne (palagend)
2020-05-08 10:31:06 Arthur Dayne watcher (Ubuntu Groovy): assignee Arthur Dayne (palagend)
2020-05-08 11:22:56 OpenStack Infra manila: assignee Corey Bryant (corey.bryant) suzhengwei (sue.sam)
2020-05-08 17:51:12 OpenStack Infra senlin: status In Progress Fix Released
2020-05-12 18:47:20 Corey Bryant swift (Ubuntu Focal): importance High Low
2020-05-12 18:48:24 Corey Bryant swift (Ubuntu Groovy): importance High Low
2020-05-13 19:17:37 Launchpad Janitor ironic (Ubuntu Groovy): status Triaged Fix Released
2020-05-13 19:37:05 OpenStack Infra mistral: status In Progress Fix Released
2020-05-14 03:54:48 OpenStack Infra watcher: status In Progress Fix Released
2020-05-14 04:03:44 Launchpad Janitor designate (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 17:31:27 Corey Bryant description [Impact] This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. [Test Case] 'systemctl status <service>' should not show the failure reported in [Impact]. [Regression Potential] Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential. [Impact] This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError:  Exception ignored in: <function _after_fork at 0x7f1100d075e0>  Traceback (most recent call last):    File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork      assert len(_active) == 1  AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch():      else:          eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active      # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet      # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. [Test Case] Tempest regression on a deployed openstack should pass successfully. Individual services can be tested as follows but with the scope of this issue tempest regression testing will be the goal for verifying this SRU. 'systemctl status <service>' should not show the failure reported in [Impact]. [Regression Potential] Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.
2020-05-14 19:31:51 Launchpad Janitor barbican (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 19:32:02 Launchpad Janitor networking-hyperv (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 19:32:14 Launchpad Janitor networking-bagpipe (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 19:42:19 Launchpad Janitor ironic-inspector (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 20:29:20 Launchpad Janitor manila (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 21:29:33 Launchpad Janitor python-os-ken (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 22:19:03 Launchpad Janitor networking-sfc (Ubuntu Groovy): status Triaged Fix Released
2020-05-14 23:11:30 Launchpad Janitor python-oslo.service (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 14:43:14 Launchpad Janitor networking-mlnx (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 16:18:07 Launchpad Janitor mistral (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 17:04:44 Launchpad Janitor senlin (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 17:04:58 Launchpad Janitor watcher (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 17:05:29 Launchpad Janitor openstack-trove (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 19:13:12 Corey Bryant murano (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 19:14:52 Corey Bryant murano-agent (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 19:16:14 Corey Bryant networking-l2gw (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 19:17:08 Corey Bryant neutron-dynamic-routing (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 20:05:50 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2020-05-15 21:33:42 Steve Langasek ironic (Ubuntu Focal): status Triaged Fix Committed
2020-05-15 21:34:36 Steve Langasek bug added subscriber SRU Verification
2020-05-15 21:34:45 Steve Langasek tags in-stable-train in-stable-ussuri py38 in-stable-train in-stable-ussuri py38 verification-needed verification-needed-focal
2020-05-15 21:44:48 Launchpad Janitor magnum (Ubuntu Groovy): status Triaged Fix Released
2020-05-15 23:24:09 OpenStack Infra manila: status In Progress Fix Released
2020-05-15 23:25:02 Steve Langasek heat (Ubuntu Focal): status Triaged Fix Committed
2020-05-15 23:31:50 Steve Langasek designate (Ubuntu Focal): status Triaged Fix Committed
2020-05-15 23:39:03 Steve Langasek barbican (Ubuntu Focal): status Triaged Fix Committed
2020-05-15 23:45:57 Steve Langasek ironic-inspector (Ubuntu Focal): status Triaged Fix Committed
2020-05-15 23:48:40 Steve Langasek magnum (Ubuntu Focal): status Triaged Fix Committed
2020-05-18 21:31:10 Brian Murray openstack-trove (Ubuntu Focal): status Triaged Fix Committed
2020-05-18 22:28:42 Brian Murray masakari (Ubuntu Focal): status Triaged Fix Committed
2020-05-18 22:34:45 Brian Murray networking-mlnx (Ubuntu Focal): status Triaged Fix Committed
2020-05-18 22:57:53 Brian Murray networking-bagpipe (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 01:40:39 Brian Murray python-os-ken (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 01:46:34 Brian Murray python-oslo.service (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 15:12:01 Brian Murray networking-hyperv (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 15:59:53 Brian Murray networking-sfc (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 16:34:02 Brian Murray manila (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 16:43:26 Brian Murray mistral (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 17:10:54 Brian Murray senlin (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 17:14:31 Corey Bryant masakari (Ubuntu Focal): status Fix Committed Triaged
2020-05-19 17:18:10 Brian Murray watcher (Ubuntu Focal): status Triaged Fix Committed
2020-05-19 21:07:39 Launchpad Janitor masakari (Ubuntu Groovy): status Triaged Fix Released
2020-05-19 21:53:49 Brian Murray masakari (Ubuntu Focal): status Triaged Fix Committed
2020-05-27 01:58:43 OpenStack Infra trove: assignee Corey Bryant (corey.bryant) Lingxian Kong (kong)
2020-05-28 12:30:35 Corey Bryant tags in-stable-train in-stable-ussuri py38 verification-needed verification-needed-focal in-stable-train in-stable-ussuri py38 verification-done verification-done-focal
2020-05-28 18:27:34 Corey Bryant cinder (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:28:48 Corey Bryant murano (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:42:35 Corey Bryant murano-agent (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:43:43 Corey Bryant networking-l2gw (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:47:42 Corey Bryant neutron (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:51:15 Corey Bryant neutron-dynamic-routing (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:52:19 Corey Bryant sahara (Ubuntu Focal): status Triaged Fix Committed
2020-05-28 18:52:44 Corey Bryant sahara (Ubuntu Groovy): status Triaged Fix Released
2020-05-28 21:14:23 Launchpad Janitor heat (Ubuntu Groovy): status Triaged Fix Released
2020-06-02 17:40:13 OpenStack Infra designate: status In Progress Fix Released
2020-06-02 17:43:43 Launchpad Janitor ironic (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:43:57 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2020-06-02 17:48:11 Launchpad Janitor heat (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:48:31 Launchpad Janitor magnum (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:58:27 Launchpad Janitor designate (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:58:40 Launchpad Janitor ironic-inspector (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:58:46 Launchpad Janitor barbican (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:59:32 Launchpad Janitor openstack-trove (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:59:38 Launchpad Janitor networking-mlnx (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 17:59:50 Launchpad Janitor networking-bagpipe (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:09:05 Launchpad Janitor python-oslo.service (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:09:18 Launchpad Janitor networking-hyperv (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:11:30 Launchpad Janitor networking-sfc (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:11:43 Launchpad Janitor mistral (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:11:55 Launchpad Janitor senlin (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:12:13 Launchpad Janitor masakari (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:19:21 Launchpad Janitor python-os-ken (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:21:54 Launchpad Janitor manila (Ubuntu Focal): status Fix Committed Fix Released
2020-06-02 18:22:14 Launchpad Janitor watcher (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:23:50 Corey Bryant neutron (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:25:22 Corey Bryant neutron-dynamic-routing (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:25:52 Corey Bryant sahara (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:27:11 Corey Bryant networking-l2gw (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:28:39 Corey Bryant murano-agent (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:29:24 Corey Bryant murano (Ubuntu Focal): status Fix Committed Fix Released
2020-06-03 13:30:33 Corey Bryant cinder (Ubuntu Focal): status Fix Committed Fix Released
2020-06-04 09:18:18 Bernard Cafarelli tags in-stable-train in-stable-ussuri py38 verification-done verification-done-focal in-stable-train in-stable-ussuri neutron-proactive-backport-potential py38 verification-done verification-done-focal
2020-07-18 12:08:44 OpenStack Infra trove: status In Progress Fix Released
2020-07-20 08:09:55 Bernard Cafarelli tags in-stable-train in-stable-ussuri neutron-proactive-backport-potential py38 verification-done verification-done-focal in-stable-train in-stable-ussuri py38 verification-done verification-done-focal
2020-09-22 07:43:09 OpenStack Infra masakari: status In Progress Fix Released
2021-07-21 11:03:35 OpenStack Infra networking-mlnx: status In Progress Fix Released