Cinder

periodic_interval only works at a minimum of 60 seconds for periodic_tasks

Bug #1695018 reported by Andreas Karis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Sean McGinnis
OpenStack Shared File Systems Service (Manila)
Opinion
Undecided
Unassigned

Bug Description

periodic_interval only works at a minimum of 60 seconds for periodic_tasks
This bug report is a bit the contrary of https://bugs.launchpad.net/nova/+bug/1319232 --- tasks don't run at the user specified frequency for anything < 60 seconds. periodic_interval only works for >= 60 second intervals.

I found this one here that explains periodic_task:
http://nickelchen.com/periodic_task/

Default interval for periodic_tasks is DEFAULT_INTERVAL = 60.0 in:
/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py
~~~
def periodic_task(*args, **kwargs):
    """Decorator to indicate that a method is a periodic task.

    This decorator can be used in two ways:

        1. Without arguments '@periodic_task', this will be run on the default
           interval of 60 seconds.

        2. With arguments:
           @periodic_task(spacing=N [, run_immediately=[True|False]]
           [, name=[None|"string"])
           this will be run on approximately every N seconds. If this number is
           negative the periodic task will be disabled. If the run_immediately
           argument is provided and has a value of 'True', the first run of the
           task will be shortly after task scheduler starts. If
           run_immediately is omitted or set to 'False', the first time the
           task runs will be approximately N seconds after the task scheduler
           starts. If name is not provided, __name__ of function is used.
~~~

I then identified the way for how to set spacing=7 in the code without
a change to oslo, but to cinder:
~~~
    @periodic_task.periodic_task(spacing=7)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

At the same time, I had set
~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 1
~~~

~~~
[root@overcloud-controller-1 ~]# tail -f /var/log/cinder/*.log | grep
'Notifying Schedulers of'
2017-06-01 14:17:43.485 728707 DEBUG cinder.manager
[req-9bb3127a-7224-4bc5-831f-3087eb80079d - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:18:55.573 754824 DEBUG cinder.manager
[req-edf8d804-587e-4eb5-a533-3d02a71c0f70 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:19:01.136 754824 DEBUG cinder.manager
[req-506d4d24-7249-4311-96da-a51fc14563ab - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:19:08.140 754824 DEBUG cinder.manager
[req-506d4d24-7249-4311-96da-a51fc14563ab - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
^C
~~~
Every 7 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=7)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 60
~~~

~~~
2017-06-01 14:21:10.565 762982 DEBUG cinder.manager
[req-9bed93b0-c29f-49f9-903b-a4d1d4b6dee8 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:21:43.106 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:22:43.105 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:23:43.109 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:24:43.107 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:25:43.108 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:26:43.107 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:27:43.105 762982 DEBUG cinder.manager
[req-c9129a1f-c00c-410d-a7bc-df942156d8f7 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
~~~
Every 60 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=7)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 20
~~~

~~~
2017-06-01 14:30:17.177 794013 DEBUG cinder.manager
[req-d33cae0c-c4c4-4bbc-b3ce-a56d53fe45eb - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:30:37.178 794013 DEBUG cinder.manager
[req-d33cae0c-c4c4-4bbc-b3ce-a56d53fe45eb - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:30:57.180 794013 DEBUG cinder.manager
[req-d33cae0c-c4c4-4bbc-b3ce-a56d53fe45eb - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
~~~
Every 20 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=5)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 10
~~~
~~~
2017-06-01 14:34:37.227 794013 DEBUG cinder.manager
[req-d33cae0c-c4c4-4bbc-b3ce-a56d53fe45eb - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:01.675 812757 DEBUG cinder.manager
[req-c908e529-5062-43e8-abdb-002151906961 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:15.247 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:25.253 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:35.254 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:45.260 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:35:55.263 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:36:05.266 812757 DEBUG cinder.manager
[req-7ca46b1e-24e3-4254-a488-f1b86fe05059 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~
Every 10 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=5)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 3
~~~
~~~
2017-06-01 14:39:10.412 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:39:13.411 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:39:19.415 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:39:25.415 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:39:28.413 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:39:34.417 820628 DEBUG cinder.manager
[req-a9cb98c3-926c-4057-bbb4-a896e71e6d6b - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~
Every 3 and every 6 seconds ????

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=5)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 5
~~~
~~~
2017-06-01 14:42:03.285 836505 DEBUG cinder.manager
[req-fb8dc561-c86d-4012-83e9-8a4f409f4f9e - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:42:08.289 836505 DEBUG cinder.manager
[req-fb8dc561-c86d-4012-83e9-8a4f409f4f9e - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:42:13.292 836505 DEBUG cinder.manager
[req-fb8dc561-c86d-4012-83e9-8a4f409f4f9e - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

2017-06-01 14:42:18.296 836505 DEBUG cinder.manager
[req-fb8dc561-c86d-4012-83e9-8a4f409f4f9e - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~
Every 5 seconds

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=60)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 1
~~~
2017-06-01 14:43:27.188 843269 DEBUG cinder.manager
[req-d5dfe1a2-5871-44c1-ba5c-c2c0d558d3bc - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

2017-06-01 14:44:24.995 843269 DEBUG cinder.manager
[req-e419755a-77bc-4460-950e-5f7a69160b54 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

2017-06-01 14:45:25.034 843269 DEBUG cinder.manager
[req-e419755a-77bc-4460-950e-5f7a69160b54 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~
Every 60 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 1
~~~
2017-06-01 14:47:32.624 858210 DEBUG cinder.manager
[req-df539f1d-c2a0-416c-9f88-157b92a342f2 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:48:31.180 858210 DEBUG cinder.manager
[req-3c9c2291-3d85-4757-bdb7-2f5e35452d77 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:49:33.221 858210 DEBUG cinder.manager
[req-3c9c2291-3d85-4757-bdb7-2f5e35452d77 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
~~~
Every 60 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 90
~~~
~~~
2017-06-01 14:52:51.813 869318 DEBUG cinder.manager
[req-e9c7247c-4c11-4a57-be8c-8521c8ecc6b2 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:54:21.812 869318 DEBUG cinder.manager
[req-e9c7247c-4c11-4a57-be8c-8521c8ecc6b2 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:55:51.810 869318 DEBUG cinder.manager
[req-e9c7247c-4c11-4a57-be8c-8521c8ecc6b2 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~

Every 90 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=1)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 17
~~~
~~~

2017-06-01 14:57:31.063 893811 DEBUG cinder.manager
[req-13d6e949-e437-4982-b27d-a8bd1cc3ae58 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:57:48.065 893811 DEBUG cinder.manager
[req-13d6e949-e437-4982-b27d-a8bd1cc3ae58 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 14:58:05.072 893811 DEBUG cinder.manager
[req-13d6e949-e437-4982-b27d-a8bd1cc3ae58 - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~

Every 17 seconds!

++++++++++++++++++++++++

Another test:
~~~
    @periodic_task.periodic_task(spacing=CONF.periodic_interval)
    def _publish_service_capabilities(self, context):
        """Pass data back to the scheduler at a periodic interval."""
        if self.last_capabilities:
            LOG.debug('Notifying Schedulers of capabilities ...')
            self.scheduler_rpcapi.update_service_capabilities(
                context,
                self.service_name,
                self.host,
                self.last_capabilities)
~~~

~~~
# Interval, in seconds, between running periodic tasks (integer value)
periodic_interval = 4
~~~
~~~
2017-06-01 15:00:27.939 902534 DEBUG cinder.manager
[req-15b9009c-037d-48a4-9675-a123193dd05c - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 15:00:31.940 902534 DEBUG cinder.manager
[req-15b9009c-037d-48a4-9675-a123193dd05c - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 15:00:35.945 902534 DEBUG cinder.manager
[req-15b9009c-037d-48a4-9675-a123193dd05c - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165
2017-06-01 15:00:39.947 902534 DEBUG cinder.manager
[req-15b9009c-037d-48a4-9675-a123193dd05c - - - - -] Notifying
Schedulers of capabilities ... _publish_service_capabilities
/usr/lib/python2.7/site-packages/cinder/manager.py:165

~~~

Every 4 seconds!

++++++++++++++++++++++++++++

The result is that this is run at the max(spacing,periodic_interval)
in most cases (I can't explain the 5 spacing, 3 periodic_interval case
yet), meaning that with the defaults set, this will run every 60
seconds and we can only force this to run at longer intervals than 60
seconds, not at shorter intervals.

Which isn't at all what the description of periodic_interval is saying.

The solution, as suggested by a friend:
@periodic_task.periodic_task(spacing=CONF.periodic_interval)

Alternatively, this also seems to work
@periodic_task.periodic_task(spacing=1)

Revision history for this message
Andreas Karis (akaris) wrote :
Download full text (4.2 KiB)

Also, note that the actual code that's starting and setting the
interval is here:
/usr/lib/python2.7/site-packages/cinder/service.py
~~~
from oslo_service import loopingcall
(...)
        if self.periodic_interval:
            if self.periodic_fuzzy_delay:
                initial_delay = random.randint(0, self.periodic_fuzzy_delay)
            else:
                initial_delay = None

            periodic = loopingcall.FixedIntervalLoopingCall(
                self.periodic_tasks)
            periodic.start(interval=self.periodic_interval,
                           initial_delay=initial_delay)
            self.timers.append(periodic)
~~~

/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py:class
FixedIntervalLoopingCall(LoopingCallBase):
~~~
class FixedIntervalLoopingCall(LoopingCallBase):
    """A fixed interval looping call."""

    _RUN_ONLY_ONE_MESSAGE = _("A fixed interval looping call can only run"
                              " one function at a time")

    _KIND = _('Fixed interval looping call')

    def start(self, interval, initial_delay=None, stop_on_exception=True):
        def _idle_for(result, elapsed):
            delay = round(elapsed - interval, 2)
            if delay > 0:
                func_name = reflection.get_callable_name(self.f)
                LOG.warning(_LW('Function %(func_name)r run outlasted '
                                'interval by %(delay).2f sec'),
                            {'func_name': func_name, 'delay': delay})
            return -delay if delay < 0 else 0
        return self._start(_idle_for, initial_delay=initial_delay,
                           stop_on_exception=stop_on_exception)
~~~

And from LoopingCallBase
~~~
    def _start(self, idle_for, initial_delay=None, stop_on_exception=True):
        """Start the looping

        :param idle_for: Callable that takes two positional arguments, returns
                         how long to idle for. The first positional argument is
                         the last result from the function being looped and the
                         second positional argument is the time it took to
                         calculate that result.
        :param initial_delay: How long to delay before starting the looping.
                              Value is in seconds.
        :param stop_on_exception: Whether to stop if an exception occurs.
        :returns: eventlet event instance
        """
        if self._thread is not None:
            raise RuntimeError(self._RUN_ONLY_ONE_MESSAGE)
        self._running = True
        self.done = event.Event()
        self._thread = greenthread.spawn(
            self._run_loop, idle_for,
            initial_delay=initial_delay, stop_on_exception=stop_on_exception)
        self._thread.link(self._on_done)
        return self.done
(...)
   def _run_loop(self, idle_for_func,
                  initial_delay=None, stop_on_exception=True):
        kind = self._KIND
        func_name = reflection.get_callable_name(self.f)
        func = self.f if stop_on_exception else _safe_wrapper(self.f, kind,
                                                              func_name)
        if initial_delay:
            greenthrea...

Read more...

Revision history for this message
Andreas Karis (akaris) wrote :
Download full text (3.5 KiB)

/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py
~~~
    def run_periodic_tasks(self, context, raise_on_error=False):
        """Tasks to be run at a periodic interval."""
        idle_for = DEFAULT_INTERVAL
        for task_name, task in self._periodic_tasks:
            if (task._periodic_external_ok and not
               self.conf.run_external_periodic_tasks):
                continue
            cls_name = reflection.get_class_name(self, fully_qualified=False)
            full_task_name = '.'.join([cls_name, task_name])

            spacing = self._periodic_spacing[task_name]
            last_run = self._periodic_last_run[task_name]

            # Check if due, if not skip
            idle_for = min(idle_for, spacing)
            if last_run is not None:
                delta = last_run + spacing - now()
                if delta > 0:
                    idle_for = min(idle_for, delta)
                    continue

            LOG.debug("Running periodic task %(full_task_name)s",
                      {"full_task_name": full_task_name})
            self._periodic_last_run[task_name] = _nearest_boundary(
                last_run, spacing)

            try:
                task(self, context)
            except Exception:
                if raise_on_error:
                    raise
                LOG.exception(_LE("Error during %(full_task_name)s"),
                              {"full_task_name": full_task_name})
            time.sleep(0)

        return idle_for
~~~

Sets a minimum interval of spacing for periodic task

~~~
[root@overcloud-controller-1 ~]# grep run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/* -R
/usr/lib/python2.7/site-packages/cinder/manager.py: return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
~~~
~~~
class Manager(base.Base, PeriodicTasks):
    # Set RPC API version to 1.0 by default.
    RPC_API_VERSION = '1.0'

    target = messaging.Target(version=RPC_API_VERSION)

    def __init__(self, host=None, db_driver=None, cluster=None):
        if not host:
            host = CONF.host
        self.host = host
        self.cluster = cluster
        self.additional_endpoints = []
        super(Manager, self).__init__(db_driver)

    def periodic_tasks(self, context, raise_on_error=False):
        """Tasks to be run at a periodic interval."""
        return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
~~~

/usr/lib/python2.7/site-packages/cinder/service.py
~~~
        if self.periodic_interval:
            if self.periodic_fuzzy_delay:
                initial_delay = random.randint(0, self.periodic_fuzzy_delay)
            else:
                initial_delay = None

            periodic = loopingcall.FixedIntervalLoopingCall(
                self.periodic_tasks)
            periodic.start(interval=self.periodic_interval,
                           initial_delay=initial_delay)
            self.timers.append(periodic)
(...)
    def periodic_tasks(self, raise_on_error=False):
        """Tasks to be run at a periodic interval."""
        ctxt = context.get_admin_context()
        self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
~~~

If I und...

Read more...

Revision history for this message
Andreas Karis (akaris) wrote :
Download full text (5.8 KiB)

It also looks as if nova was getting this right:

[root@overcloud-controller-1 ~]# grep periodic_task /usr/lib/python2.7/site-packages/nova* -R
/usr/lib/python2.7/site-packages/nova/manager.py:from oslo_service import periodic_task
/usr/lib/python2.7/site-packages/nova/manager.py:class PeriodicTasks(periodic_task.PeriodicTasks):
/usr/lib/python2.7/site-packages/nova/manager.py: def periodic_tasks(self, context, raise_on_error=False):
/usr/lib/python2.7/site-packages/nova/manager.py: return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
/usr/lib/python2.7/site-packages/nova/cells/manager.py:from oslo_service import periodic_task
/usr/lib/python2.7/site-packages/nova/cells/manager.py: @periodic_task.periodic_task
/usr/lib/python2.7/site-packages/nova/cells/manager.py: @periodic_task.periodic_task
Binary file /usr/lib/python2.7/site-packages/nova/cells/manager.pyo matches
Binary file /usr/lib/python2.7/site-packages/nova/cells/manager.pyc matches
Binary file /usr/lib/python2.7/site-packages/nova/service.pyo matches
Binary file /usr/lib/python2.7/site-packages/nova/manager.pyo matches
Binary file /usr/lib/python2.7/site-packages/nova/service.pyc matches
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/driver.pyo matches
/usr/lib/python2.7/site-packages/nova/scheduler/manager.py:from oslo_service import periodic_task
/usr/lib/python2.7/site-packages/nova/scheduler/manager.py: @periodic_task.periodic_task
/usr/lib/python2.7/site-packages/nova/scheduler/manager.py: @periodic_task.periodic_task(spacing=CONF.scheduler_driver_task_period,
/usr/lib/python2.7/site-packages/nova/scheduler/manager.py: def _run_periodic_tasks(self, context):
/usr/lib/python2.7/site-packages/nova/scheduler/manager.py: self.driver.run_periodic_tasks(context)
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/caching_scheduler.pyo matches
/usr/lib/python2.7/site-packages/nova/scheduler/caching_scheduler.py: def run_periodic_tasks(self, context):
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/manager.pyo matches
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/driver.pyc matches
/usr/lib/python2.7/site-packages/nova/scheduler/driver.py: def run_periodic_tasks(self, context):
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/caching_scheduler.pyc matches
Binary file /usr/lib/python2.7/site-packages/nova/scheduler/manager.pyc matches
/usr/lib/python2.7/site-packages/nova/rpc.py:from oslo_service import periodic_task
/usr/lib/python2.7/site-packages/nova/rpc.py:class ClientRouter(periodic_task.PeriodicTasks):
/usr/lib/python2.7/site-packages/nova/rpc.py: self.run_periodic_tasks(nova.context.RequestContext(overwrite=False))
/usr/lib/python2.7/site-packages/nova/rpc.py: @periodic_task.periodic_task
Binary file /usr/lib/python2.7/site-packages/nova/rpc.pyo matches
/usr/lib/python2.7/site-packages/nova/compute/manager.py:from oslo_service import periodic_task
/usr/lib/python2.7/site-packages/nova/compute/manager.py: @periodic_task.periodic_task
/usr/lib/python2.7/site-packages/nova/compute/manager.py: @periodic_task.periodic_task(spacing=CONF.scheduler_instance_...

Read more...

Changed in cinder:
assignee: nobody → Andreas Karis (akaris)
status: New → Confirmed
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/469936

Goutham Pacha Ravi (gouthamr)
Changed in manila:
status: New → Opinion
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote : Bug Assignee Expired

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Andreas Karis (akaris) → nobody
status: In Progress → Triaged
OpenStack Infra (hudson-openstack)
Changed in cinder:
assignee: nobody → Sean McGinnis (sean-mcginnis)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/469936
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=694d8f092a277350a8ca69b2b4229a865f3ba380
Submitter: Zuul
Branch: master

commit 694d8f092a277350a8ca69b2b4229a865f3ba380
Author: Andreas Karis <email address hidden>
Date: Thu Jun 1 12:03:57 2017 -0400

    Enable service capabilities update at less than 60 seconds

    periodic_interval only works at a minimum of 60 seconds
    for periodic_tasks that update service capabilities
    Make it possible for service capabilities updates to run
    at intervals < 60 seconds as defined in parameter
    periodic_interval in cinder.conf

    Change-Id: I0f78aa7dc04dc0d74fed7c2c84d3442f2993258d
    Closes-Bug: #1695018

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.