Seen in heat engine:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 783, in _action_recorder
self.state_set(action, self.FAILED, msg)
File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2015, in state_set
self._add_event(action, status, reason)
File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1769, in _add_event
self.stack.dispatch_event(ev)
File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 907, in dispatch_event
ev.as_dict())
File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 143, in start
func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 96, in add_thread
th = Thread(gt, self)
File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 46, in __init__
self.thread.link(_on_thread_done, group, self)
File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 197, in link
self._resolve_links()
File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 232, in _resolve_links
f(self, *ca, **ckw)
File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 34, in _on_thread_done
group.thread_done(thread)
File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 101, in thread_done
self.threads.remove(thread)
ValueError: list.remove(x): x not in list
It looks like a race condition: if the callback is fast enough, the thread can exit before the link, which is not accounted for.
See https:/ /github. com/eventlet/ eventlet/ blob/master/ eventlet/ greenthread. py#L200 for the reason. Links can be resolved right away, but we only add the thread afterwards.