list.remove(x): x not in list when creating a thread

Bug #1662939 reported by Thomas Herve on 2017-02-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
High
Thomas Herve

Bug Description

Seen in heat engine:

Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 783, in _action_recorder
     self.state_set(action, self.FAILED, msg)
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2015, in state_set
     self._add_event(action, status, reason)
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1769, in _add_event
     self.stack.dispatch_event(ev)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 907, in dispatch_event
     ev.as_dict())
   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 143, in start
     func, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 96, in add_thread
     th = Thread(gt, self)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 46, in __init__
     self.thread.link(_on_thread_done, group, self)
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 197, in link
     self._resolve_links()
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 232, in _resolve_links
     f(self, *ca, **ckw)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 34, in _on_thread_done
     group.thread_done(thread)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 101, in thread_done
     self.threads.remove(thread)
 ValueError: list.remove(x): x not in list

It looks like a race condition: if the callback is fast enough, the thread can exit before the link, which is not accounted for.

Thomas Herve (therve) on 2017-02-08
Changed in oslo.service:
assignee: nobody → Thomas Herve (therve)
Thomas Herve (therve) wrote :

See https://github.com/eventlet/eventlet/blob/master/eventlet/greenthread.py#L200 for the reason. Links can be resolved right away, but we only add the thread afterwards.

Thomas Herve (therve) on 2017-02-09
Changed in oslo.service:
status: New → In Progress
tags: added: ocata-backport-potential
Changed in oslo.service:
importance: Undecided → High

This issue was fixed in the openstack/oslo.service 1.20.0 release.

Changed in oslo.service:
status: In Progress → Fix Released
Thomas Herve (therve) on 2017-05-30
tags: added: newton-backport-potential

This issue was fixed in the openstack/oslo.service 1.19.1 release.

This issue was fixed in the openstack/oslo.service 1.16.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers