scheduler should implement stepoff delay

[gordon chung]

currently, the scheduler process will grab a set of metrics and distribute it to the processing workers. if the scheduler is grabbing faster than workers finish, its next set may overlap partially with previous set of tasks. the existing safety mechanism will remove metrics already queued by previous set but if 3rd query still overlaps 1st query set, it will requeue the task which will eventually be a NOOP.

this isn't a terrible scenario as we log issue and it in practice will only requeue every other schedule interval (ie, 1st, 3rd, 5th, etc...) but it'd be nice to maybe detect overlap and delay the next scheduling query based on some logic?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/449316

[OpenStack Infra (hudson-openstack)]

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/449316

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441389
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=9889bd04f5ae9949c739ab656771aa5b8c1bab95
Submitter: Jenkins
Branch: master

commit 9889bd04f5ae9949c739ab656771aa5b8c1bab95
Author: gord chung <email address hidden>
Date: Fri Mar 3 19:31:20 2017 +0000

    push incoming into different sacks

    having everything in one giant folder/bucket/container/object is bad
    because:
    - does not allow for good distribution of backend driver.
    - makes it hella hard to cleanly split work across multiple metricd
    - starves metrics from being processed.

    we used "sacks" to avoid naming same as driver paradigms.

    driver implementation will be done individually so this does nothing but
    pretend to lock multiple buckets when it's actually just one.

    Related-Bug: #1629420
    Related-Bug: #1623263
    Related-Bug: #1620674
    Change-Id: Icc32d918fe55416385122470c47d60ddbb30dd34