list.remove(x): x not in list when creating a thread

Bug #1662939 reported by Thomas Herve
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.service
Fix Released
High
Thomas Herve

Bug Description

Seen in heat engine:

Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 783, in _action_recorder
     self.state_set(action, self.FAILED, msg)
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2015, in state_set
     self._add_event(action, status, reason)
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1769, in _add_event
     self.stack.dispatch_event(ev)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 907, in dispatch_event
     ev.as_dict())
   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 143, in start
     func, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 96, in add_thread
     th = Thread(gt, self)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 46, in __init__
     self.thread.link(_on_thread_done, group, self)
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 197, in link
     self._resolve_links()
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 232, in _resolve_links
     f(self, *ca, **ckw)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 34, in _on_thread_done
     group.thread_done(thread)
   File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 101, in thread_done
     self.threads.remove(thread)
 ValueError: list.remove(x): x not in list

It looks like a race condition: if the callback is fast enough, the thread can exit before the link, which is not accounted for.

Thomas Herve (therve)
Changed in oslo.service:
assignee: nobody → Thomas Herve (therve)
Revision history for this message
Thomas Herve (therve) wrote :

See https://github.com/eventlet/eventlet/blob/master/eventlet/greenthread.py#L200 for the reason. Links can be resolved right away, but we only add the thread afterwards.

Revision history for this message
Thomas Herve (therve) wrote :
Thomas Herve (therve)
Changed in oslo.service:
status: New → In Progress
tags: added: ocata-backport-potential
Changed in oslo.service:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.service 1.20.0

This issue was fixed in the openstack/oslo.service 1.20.0 release.

Changed in oslo.service:
status: In Progress → Fix Released
Thomas Herve (therve)
tags: added: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.service 1.19.1

This issue was fixed in the openstack/oslo.service 1.19.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.service 1.16.1

This issue was fixed in the openstack/oslo.service 1.16.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.