Comment 4 for bug 1417281

Reviewed: https://review.openstack.org/234257
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=7f76ad62aa68d8d71fe2773ae8ad817103051b88
Submitter: Jenkins
Branch: master

commit 7f76ad62aa68d8d71fe2773ae8ad817103051b88
Author: Kun Huang <email address hidden>
Date: Tue Oct 13 22:02:15 2015 +0800

    Fix random miss in comsumer/publisher broker

    The error in bug report is comfirmed but reproduced randomly. Runing [0]
    100 times and you could get about 3 failures on average. The reason is
    that publisher finished between L39 and L40 here [1].

    Before this patch, we are starting comsumers first and then publisher.
    Before publisher doing publish(), comsumer threads are in while-true
    loop because the thread event is_published is not setted. At the moment
    publish work has been done, which means the is_published is setted,
    each comsumer thread could be in any position of the while-true loop. If
    it happens between L39 and L40, the broker will miss the job. If there
    is only one comsumer here (it is in our test case), the miss happens.

    To clarify it:

    39 while True:
    40 if not queue:
            <<< this moment, publisher finished >>>
            <<< which means queue is not empty >>>
            <<< and is_published is setted. >>>
    41 if is_published.isSet():
    42 break
    43 time.sleep(0.1)
    44 continue
    45 else:
    46 try:
    47 args = queue.popleft()
    48 except IndexError:
    49 # consumed by other thread
    50 continue

    To fix this is very simple: do publish job BEFORE the comsumers start
    and remove the unrequired is_published :)

    [0] tox -e py27
    tests.unit.plugins.openstack.context.keystone.test_users.UserGeneratorTestCase.test_users_and_tenants_in_context
    [1]
    https://github.com/openstack/rally/blob/master/rally/common/broker.py#L39-L40

    Change-Id: Ieab25e531382e5c7832eeaecb962d6139a654b96
    Closes-bug: #1417281