Periodic tasks run too frequently
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Undecided
|
Matt Riedemann | ||
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Matt Riedemann | ||
neutron |
Fix Released
|
Undecided
|
Matt Riedemann | ||
oslo-incubator |
Fix Released
|
Undecided
|
Matt Riedemann |
Bug Description
Each periodic task can have a "spacing", which defines the minimum amount of time between executions of that task. For example, a task with periodic_
As a result of these "complications", we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value.
The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect.
My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/
# A periodic spacing of zero indicates that this task should
# be run every pass
if task._periodic_
to:
# A periodic spacing of zero indicates that this task should
# be run at the default interval
if task._periodic_
The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular.
Changed in nova: | |
status: | Confirmed → Fix Committed |
assignee: | nobody → Matt Riedemann (mriedem) |
milestone: | none → juno-1 |
Changed in oslo: | |
milestone: | none → juno-1 |
Changed in neutron: | |
status: | New → In Progress |
assignee: | nobody → Matt Riedemann (mriedem) |
Changed in cinder: | |
status: | New → In Progress |
assignee: | nobody → Matt Riedemann (mriedem) |
Changed in neutron: | |
status: | In Progress → Fix Committed |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in neutron: | |
milestone: | none → juno-1 |
status: | Fix Committed → Fix Released |
Changed in oslo: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | juno-1 → 2014.2 |
Changed in neutron: | |
milestone: | juno-1 → 2014.2 |
There was also some related discussion in the mailing list here:
https:/ /www.mail- archive. com/openstack- dev%40lists. openstack. org/msg15922. html