Periodic tasks run too frequently

Bug #1319232 reported by Jeremy Arnold
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Matt Riedemann
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
neutron
Fix Released
Undecided
Matt Riedemann
oslo-incubator
Fix Released
Undecided
Matt Riedemann

Bug Description

Each periodic task can have a "spacing", which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as "every 60 seconds", but in reality it's more complicated than that.

As a result of these "complications", we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value.

The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect.

My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code:

        # A periodic spacing of zero indicates that this task should
        # be run every pass
        if task._periodic_spacing == 0:
            task._periodic_spacing = None

to:
        # A periodic spacing of zero indicates that this task should
        # be run at the default interval
        if task._periodic_spacing == 0:
            task._periodic_spacing = DEFAULT_INTERVAL

The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular.

Tags: compute oslo
Revision history for this message
Matt Riedemann (mriedem) wrote :

There was also some related discussion in the mailing list here:

https://www.mail-archive.com/openstack-dev%40lists.openstack.org/msg15922.html

tags: added: compute oslo
Changed in nova:
status: New → Confirmed
importance: Undecided → High
importance: High → Medium
Revision history for this message
Matt Riedemann (mriedem) wrote :

Related nova change here: https://review.openstack.org/#/c/72684/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/93767

Changed in oslo:
assignee: nobody → Matt Riedemann (mriedem)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/93767
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=c63fd5a439f22cf32ebea951e6f3b150374004f9
Submitter: Jenkins
Branch: master

commit c63fd5a439f22cf32ebea951e6f3b150374004f9
Author: Matt Riedemann <email address hidden>
Date: Thu May 15 07:51:43 2014 -0700

    Make unspecified periodic spaced tasks run on default interval

    When there are multiple tasks with a defined spacing, they can get
    somewhat staggered and force the periodic task processor to run
    additional iterations. Since tasks with no spacing run every time the
    periodic task processor runs, they get run more frequently than one
    would expect.

    Some may argue that this change in behavior could cause some task to
    behave differently than it used to. However, there was never any
    guarantee that the task would run more often than every 60 seconds, and
    in many cases the tasks may already run less frequently than that (due
    to other long-running tasks). So this change should not introduce any
    new issues related to the timing of task execution; it would only serve
    to make the timing more regular.

    We should also make the default interval configurable but that's
    reserved for a separate change.

    DocImpact: periodic tasks without a specific spacing interval will now
    run on the default interval (currently every 60 seconds) rather than
    whenever the task processor runs.

    UpgradeImpact: see above; periodic tasks without a specific spacing
    interval will now run on the default interval which may be less
    often than some tasks run now, but with this change they will run
    on a more consistent interval.

    Closes-Bug: #1319232
    Related-Bug: #1276203
    Related-Bug: #1272830

    Change-Id: I4dd20f68175165d311fa4d3f459c86f2dcac911b

Changed in oslo:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/93520
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a324daab957d75b45ae2374d30b8ab39be587d7c
Submitter: Jenkins
Branch: master

commit a324daab957d75b45ae2374d30b8ab39be587d7c
Author: Matt Riedemann <email address hidden>
Date: Tue May 13 15:10:03 2014 -0700

    Sync periodic_task from oslo-incubator

    We have not updated periodic_task in over six months and we should have
    commit 47c9d60657be4ad58afa3c7e3ef88885a595c4c3 in nova.

    Note that this does not include the gettextutils and log dependencies
    since there are not functional changes in those modules needed for the
    periodic_task changes synced in *and* more importantly, the changes
    to gettextutils and log require pervasive changes to nova which should
    happen when nova integrates with the oslo-i18n library for
    blueprint i18n--messages.

    Further note that this does not include jsonutils due to some
    issues introduced with a change for python 2.6 that impacts some of
    the unit tests in Nova and how strings are encoded with simplejson.
    The details for that issue are in bug 1314129. The jsonutils changes
    are not related to the periodic_task changes being synced in so the
    dependency is not functionally required.

    periodic_task:
    c63fd5a Make unspecified periodic spaced tasks run on default interval
    f0dd798 Remove rendundant parentheses of cfg help strings
    fcf517d Update oslo log messages with translation domains
    051b9f3 Refactor unnecessary arithmetic ops in periodic_task
    674cdaf Refactor if logic in periodic_task
    b6b82c5 Use timestamp in periodic tasks
    47c9d60 Don't share periodic_task instance data in a class attr
    8b2b0b7 Use hacking import_exceptions for gettextutils._
    c5a1088 Typos fix in db and periodic_task module
    12bcdb7 Remove vim header

    Related-Bug: #1319232

    Change-Id: I5534e85ae8791c7c464357f7fd57b986ff74d092

Matt Riedemann (mriedem)
Changed in nova:
status: Confirmed → Fix Committed
assignee: nobody → Matt Riedemann (mriedem)
milestone: none → juno-1
Changed in oslo:
milestone: none → juno-1
Revision history for this message
Matt Riedemann (mriedem) wrote :

Added Cinder since there are at least two periodic tasks in Cinder that don't have specific spacing set.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Added Neutron since there are periodic tasks on the agents to sync up values with the server.

Revision history for this message
Matt Riedemann (mriedem) wrote :

For neutron, specifically, L3, LBaaS, and metering agents.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/96499

Matt Riedemann (mriedem)
Changed in neutron:
status: New → In Progress
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/96512

Matt Riedemann (mriedem)
Changed in cinder:
status: New → In Progress
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/96499
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=741eb5f4d57ec8b9fe24196a635173714cd910ab
Submitter: Jenkins
Branch: master

commit 741eb5f4d57ec8b9fe24196a635173714cd910ab
Author: Matt Riedemann <email address hidden>
Date: Thu May 29 07:33:16 2014 -0700

    Sync periodic_task from oslo-incubator

    This is more or less to get commit c63fd5a from oslo into the core
    projects which have several periodic tasks. Neutron has periodic tasks
    for L3, load balancing and metering agents to sync up state with the
    server and most don't have specific spacing values set which can lead to
    non-deterministic spacing of when the tasks run.

    Note that this does not include the gettextutils and log dependencies
    since there are not functional changes in those modules needed for the
    periodic_task changes synced in *and* more importantly, the changes
    to gettextutils and log require pervasive changes to neutron which
    should happen when neutron integrates with the oslo-i18n library for
    blueprint i18n--messages.

    Further note that this does not include jsonutils due to some
    issues introduced with a change for python 2.6 that impacts how strings
    are encoded with simplejson. The details for that issue are in bug
    1314129. The jsonutils changes are not related to the periodic_task
    changes being synced in so the dependency is not functionally required.

    The LbaasAgentManager extends PeriodicTasks but wasn't calling the
    parent class init function, which was causing failures since commit
    47c9d60 changed PeriodicTasks to init _periodic_last_run, so also
    fixed that here.

    Changes:

    c63fd5a Make unspecified periodic spaced tasks run on default interval
    f0dd798 Remove rendundant parentheses of cfg help strings
    fcf517d Update oslo log messages with translation domains
    051b9f3 Refactor unnecessary arithmetic ops in periodic_task
    674cdaf Refactor if logic in periodic_task
    b6b82c5 Use timestamp in periodic tasks
    47c9d60 Don't share periodic_task instance data in a class attr
    8b2b0b7 Use hacking import_exceptions for gettextutils._

    Related-Bug: #1319232

    Change-Id: Ib23e33326129be0dd952efde263f381d5aa2457f

Matt Riedemann (mriedem)
Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.openstack.org/96512
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=132c4530713a6f404ce5ed01a9668519d69afe16
Submitter: Jenkins
Branch: master

commit 132c4530713a6f404ce5ed01a9668519d69afe16
Author: Matt Riedemann <email address hidden>
Date: Thu May 29 08:45:55 2014 -0700

    Sync periodic_task from oslo-incubator

    This is more or less to get commit c63fd5a from oslo into the core
    projects which have several periodic tasks. Cinder has periodic tasks
    to sync up volume state with the server and they don't have specific
    spacing values set which can lead to non-deterministic spacing of when
    the tasks run.

    Note that this does not include the gettextutils and log dependencies
    since there are not functional changes in those modules needed for the
    periodic_task changes synced in *and* more importantly, the changes
    to gettextutils and log require pervasive changes to cinder which
    should happen when cinder integrates with the oslo-i18n library for
    blueprint i18n--messages.

    Further note that this does not include jsonutils due to some
    issues introduced with a change for python 2.6 that impacts how strings
    are encoded with simplejson. The details for that issue are in bug
    1314129. The jsonutils changes are not related to the periodic_task
    changes being synced in so the dependency is not functionally required.

    cinder.db.base.Base is also updated otherwise multiple inheritance
    involving that class will not work, which impacts all of the classes
    that extend cinder.manager.Manager which extends both cinder.db.Base
    and PeriodicTasks, and commit 47c9d60 adds attributes to the __init__
    for PeriodicTasks. Nova had the same change with commit 5ae97ea.

    Changes:

    c63fd5a Make unspecified periodic spaced tasks run on default interval
    f0dd798 Remove rendundant parentheses of cfg help strings
    fcf517d Update oslo log messages with translation domains
    051b9f3 Refactor unnecessary arithmetic ops in periodic_task
    674cdaf Refactor if logic in periodic_task
    b6b82c5 Use timestamp in periodic tasks
    47c9d60 Don't share periodic_task instance data in a class attr
    8b2b0b7 Use hacking import_exceptions for gettextutils._
    c5a1088 Typos fix in db and periodic_task module
    12bcdb7 Remove vim header

    Related-Bug: #1319232

    Change-Id: I13e8ac83bcd9e60a8eb05ed9cdfea00b9e5fb398

Thierry Carrez (ttx)
Changed in nova:
milestone: juno-1 → 2014.2
Thierry Carrez (ttx)
Changed in neutron:
milestone: juno-1 → 2014.2
Revision history for this message
Matt Riedemann (mriedem) wrote :

Cinder was fixed back in Juno: https://review.openstack.org/#/c/96512/

Changed in cinder:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.