Don't log GreenletExit in ServiceLauncher if any thread is killed on wait()

Bug #1382573 reported by Ihar Hrachyshka
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo-incubator
Fix Released
Undecided
Ihar Hrachyshka

Bug Description

This is similar to bug 1282206 that fixed the same issue for ProcessLauncher.

The failure shows up in gate (Icehouse, Juno) as follows:

ft1.1683: tests.unit.test_service.ServiceRestartTest.test_service_restart_StringException: Traceback (most recent call last):
  File "tests/unit/test_service.py", line 252, in test_service_restart
    ready = self._spawn_service()
  File "tests/unit/test_service.py", line 244, in _spawn_service
    launcher.wait(ready_callback=ready_event.set)
  File "openstack/common/service.py", line 196, in wait
    status, signo = self._wait_for_exit_or_signal(ready_callback)
  File "openstack/common/service.py", line 182, in _wait_for_exit_or_signal
    self.stop()
  File "openstack/common/service.py", line 128, in stop
    self.services.stop()
  File "openstack/common/service.py", line 479, in stop
    self.tg.stop()
  File "openstack/common/threadgroup.py", line 125, in stop
    self._stop_threads()
  File "openstack/common/threadgroup.py", line 98, in _stop_threads
    x.stop()
  File "openstack/common/threadgroup.py", line 44, in stop
    self.thread.kill()
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 238, in kill
    return kill(self, *throw_args)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 292, in kill
    g.throw(*throw_args)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
    result = function(*args, **kwargs)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 278, in just_raise
    raise greenlet.GreenletExit()
GreenletExit
Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

traceback-1: {{{
Traceback (most recent call last):
  File "tests/unit/test_service.py", line 93, in _reap_pid
    if self.pid:
AttributeError: 'ServiceRestartTest' object has no attribute 'pid'
}}}

Traceback (most recent call last):
  File "tests/unit/test_service.py", line 252, in test_service_restart
    ready = self._spawn_service()
  File "tests/unit/test_service.py", line 244, in _spawn_service
    launcher.wait(ready_callback=ready_event.set)
  File "openstack/common/service.py", line 196, in wait
    status, signo = self._wait_for_exit_or_signal(ready_callback)
  File "openstack/common/service.py", line 182, in _wait_for_exit_or_signal
    self.stop()
  File "openstack/common/service.py", line 128, in stop
    self.services.stop()
  File "openstack/common/service.py", line 479, in stop
    self.tg.stop()
  File "openstack/common/threadgroup.py", line 125, in stop
    self._stop_threads()
  File "openstack/common/threadgroup.py", line 98, in _stop_threads
    x.stop()
  File "openstack/common/threadgroup.py", line 44, in stop
    self.thread.kill()
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 238, in kill
    return kill(self, *throw_args)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 292, in kill
    g.throw(*throw_args)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
    result = function(*args, **kwargs)
  File "/home/jenkins/workspace/gate-oslo-incubator-python26/.tox/py26/lib/python2.6/site-packages/eventlet/greenthread.py", line 278, in just_raise
    raise greenlet.GreenletExit()
GreenletExit

Logs: http://logs.openstack.org/82/129182/1/check/gate-oslo-incubator-python26/002df95/testr_results.html.gz

Changed in neutron:
assignee: nobody → Numan Siddique (numansiddique)
Changed in neutron:
status: New → In Progress
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

I've sent https://review.openstack.org/129278 to check whether this is a similar problem as in bug 1282206.

Changed in neutron:
assignee: Numan Siddique (numansiddique) → Ihar Hrachyshka (ihar-hrachyshka)
Revision history for this message
Numan Siddique (numansiddique) wrote :

I wasn't aware of it that you are working

affects: neutron → oslo-incubator
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

The problem is not about ServiceLauncher failure, but about GreenletExit exception logged on thread kill with LOG.exception() even though it's expected behaviour when killing a greenthread. As for the test failure, it's because the unit test class does not guard correctly against potential leakage of spawned pids into test environment.

So two patches are relevant here:
- don't log GreenletExit in ThreadGroup: https://review.openstack.org/130799
- test resilience: https://review.openstack.org/129278

summary: - Uncaught GreenletExit in ServiceLauncher if wait called after greenlet
- kill
+ Don't log GreenletExit in ServiceLauncher if any thread is killed on
+ wait()
Changed in oslo-incubator:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in oslo-incubator:
milestone: none → kilo-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo-incubator:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.