Cinder service.py is out of sync from incubator that causes threads hang all the time.

[Jerry Cai]

In latest Cinder component, we used ThreadGroup in cinder/openstack/common/threadgroup to add a timer, but it dead after run 1-2 times, the thread is alive but always hang there.

This is the code piece of threadgroup.py:
    def add_timer(self, interval, callback, initial_delay=None,
                  *args, **kwargs):
        pulse = loopingcall.FixedIntervalLoopingCall(callback, *args, **kwargs)
        pulse.start(interval=interval,
                    initial_delay=initial_delay)
        self.timers.append(pulse)

#What we did is create a manager:
from cinder.openstack.common import service
class MyManager(service.Service):
....
    def __init__(self):
        self.test_periodic_task()

    #add a timer
    def test_periodic_task(self):
        def test():
            LOG.error('print!')

        LOG.error('Start to print!')
        self.tg.add_timer(5, test)

The result is "print" printed twice but after that the thread is hang there, no log any more. We also tried open the backdoor_port in cinder.conf and tried to check the GreenThread status but failed to telnet on it, seems the thread is not in response.

The codes work on previous release but failed in latest cinder component, hope the members who has insight can check on this.

[Jerry Cai (caimin)]

This should be a defect from cinder.openstack.common.service.py, it should use eventlet's event instead of threading in service.py, here is the method to fix this in incubator(which is parent for nova and cinder component): https://github.com/openstack/oslo-incubator/commit/1e7007824374842bc8108dff5e5e2694e12932b2

Note that the nova has merged the latest source while cinder doesn't, that's reason why causes the cinder-volume hang there all the time.

[Jay Bryant (jsbryant)]

Looks like we have a number of things out of sync here. I am putting together the sync right now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120495
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f342c1631b619f1239dedbe3aad93c4b474c4f55
Submitter: Jenkins
Branch: master

commit f342c1631b619f1239dedbe3aad93c4b474c4f55
Author: Jay S. Bryant <email address hidden>
Date: Wed Sep 10 10:18:22 2014 -0500

    Sync service.py and its dependencies to Cinder

    Bug 1362455 made note that our sevice.py was out of sync
    from what is currently in oslo-incubator. We have been syncing
    things piece meal. It would be good to get this code synced up.

    ----------------------------------------------

    Head of oslo-incubator:

    commit 820752a10d8277464261661ca211f2c3e7282ce0
    Merge: 8ebe497 422e94b
    Author: Jenkins <email address hidden>
    Date: Wed Sep 10 14:14:28 2014 +0000

        Merge "Switch to using pbr's autodoc capability"

    -----------------------------------------------

    This commit pulls in the following changes listed by file,
    ordered from newest to oldest:

    cinder/openstack/common/eventlet_backdoor.py
    ------------------------------------------------
    90ae24bf Remove redundant default=None for config options
    fcf517d7 Update oslo log messages with translation domains

    cinder/openstack/common/jsonutils.py
    ------------------------------------------------
    d9060f69 Check for namedtuple_as_object support before using it
    ad248f66 Specify namedtuple_as_object=False when using simplejson

    cinder/openstack/common/log.py
    ------------------------------------------------
    037dee00 Set stevedore log level to WARN by default

    cinder/openstack/common/loopingcall.py
    ------------------------------------------------
    e3773930 Changes calcuation of variable delay
    ab5d5f1c Use timestamp in loopingcall
    bc48099a Log the function name of looping call
    fb4e863c Remove deprecated LoopingCall
    fcf517d7 Update oslo log messages with translation domains

    cinder/openstack/common/service.py
    ------------------------------------------------
    5f12bba4 Implement stop method in ProcessLauncher
    8c00b361 Merge "Remove duplicated "caught" message"
    f718e404 Remove duplicated "caught" message
    356ecd4d Move notification point to a better place
    e2634a75 Add missing _LI for LOG.info in service module
    0150ad75 Merge "Reap child processes gracefully if greenlet thread gets killed"
    53e1214c notify calling process we are ready to serve
    a6fec438 Reap child processes gracefully if greenlet thread gets killed
    fcf517d7 Update oslo log messages with translation domains
    1e700782 Revert "service: replace eventlet event by threading"
    0644073e Simplify launch method
    6b4d255e Merge "service: replace eventlet event by threading"
    a4f145e4 Merge "Allow configurable ProcessLauncher liveness check"

    cinder/openstack/common/threadgroup.py
    ------------------------------------------------
    5a1a0166 Make stop_timers() method public
    fdc88831 Add graceful stop function to ThreadGroup.stop
    5f8ace05 Merge "threadgroup: use threading rather than gree...