race in rally.common.broker functionality

[Boris Pavlovic]

tests.unit.benchmark.context.test_users.UserGeneratorTestCase.test_users_and_tenants_in_context

Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-rally-python26/.tox/py26/lib/python2.6/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "tests/unit/benchmark/context/test_users.py", line 217, in test_users_and_tenants_in_context
    ctx.setup()
  File "rally/common/utils.py", line 205, in wrapper
    result = f(self, *args, **kwargs)
  File "rally/benchmark/context/users.py", line 244, in setup
    msg=_("Failed to create the requested number of tenants."))
ContextSetupFailure: Unable to setup context 'users': 'Failed to create the requested number of tenants.'

[apal (yun-song)]

I also met this error when running keystone v3 test.
v2 is good but v3 fails on the same case.

[Boris Pavlovic (boris-42)]

Didn't see any failed tests for a long amount of time

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234257

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234257
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=7f76ad62aa68d8d71fe2773ae8ad817103051b88
Submitter: Jenkins
Branch: master

commit 7f76ad62aa68d8d71fe2773ae8ad817103051b88
Author: Kun Huang <email address hidden>
Date: Tue Oct 13 22:02:15 2015 +0800

    Fix random miss in comsumer/publisher broker

    The error in bug report is comfirmed but reproduced randomly. Runing [0]
    100 times and you could get about 3 failures on average. The reason is
    that publisher finished between L39 and L40 here [1].

    Before this patch, we are starting comsumers first and then publisher.
    Before publisher doing publish(), comsumer threads are in while-true
    loop because the thread event is_published is not setted. At the moment
    publish work has been done, which means the is_published is setted,
    each comsumer thread could be in any position of the while-true loop. If
    it happens between L39 and L40, the broker will miss the job. If there
    is only one comsumer here (it is in our test case), the miss happens.

    To clarify it:

    39 while True:
    40 if not queue:
            <<< this moment, publisher finished >>>
            <<< which means queue is not empty >>>
            <<< and is_published is setted. >>>
    41 if is_published.isSet():
    42 break
    43 time.sleep(0.1)
    44 continue
    45 else:
    46 try:
    47 args = queue.popleft()
    48 except IndexError:
    49 # consumed by other thread
    50 continue

    To fix this is very simple: do publish job BEFORE the comsumers start
    and remove the unrequired is_published :)

    [0] tox -e py27
    tests.unit.plugins.openstack.context.keystone.test_users.UserGeneratorTestCase.test_users_and_tenants_in_context
    [1]
    https://github.com/openstack/rally/blob/master/rally/common/broker.py#L39-L40

    Change-Id: Ieab25e531382e5c7832eeaecb962d6139a654b96
    Closes-bug: #1417281