This fixes the sporadic of tasks that would happen
under certain circumstances. What happened was that
a new worker notification would be sent to a callback
while at the same time a task submission would come in
and there would be a small race period where the task
would insert itself into the requests cache while the
callback was processing.
So to work around this the whole concept of a requests
cache was revamped and now the WBE executor just maintains
its own local dictionary of ongoing requests and accesses
it safely.
During the on_wait function that is periodically called
by kombu the previous expiry of work happens but now any
requests that are pending are matched to any new workers
that may have appeared.
This avoids the race (and ensures that even if a new
worker is found but a submission is in progress that the
duration until that submission happens will only be until
the next on_wait call happens).
Reviewed: https:/ /review. openstack. org/276536 /git.openstack. org/cgit/ openstack/ taskflow/ commit/ ?id=cea71f27998 cfa911044103fcb 8fca79b6989717
Committed: https:/
Submitter: Jenkins
Branch: master
commit cea71f27998cfa9 11044103fcb8fca 79b6989717
Author: Joshua Harlow <email address hidden>
Date: Thu Feb 4 18:09:24 2016 -0800
Fix for WBE sporadic timeout of tasks
This fixes the sporadic of tasks that would happen
under certain circumstances. What happened was that
a new worker notification would be sent to a callback
while at the same time a task submission would come in
and there would be a small race period where the task
would insert itself into the requests cache while the
callback was processing.
So to work around this the whole concept of a requests
cache was revamped and now the WBE executor just maintains
its own local dictionary of ongoing requests and accesses
it safely.
During the on_wait function that is periodically called
by kombu the previous expiry of work happens but now any
requests that are pending are matched to any new workers
that may have appeared.
This avoids the race (and ensures that even if a new
worker is found but a submission is in progress that the
duration until that submission happens will only be until
the next on_wait call happens).
Related-Bug: #1431097
Change-Id: I98b0caeedc77ab 2f7214847763ae1 eb0433d4a78