tests.test_rabbit.TestRacyWaitForReply.test_send_receive race fail in the gate

[Matt Riedemann]

http://logs.openstack.org/72/94972/1/gate/gate-oslo.messaging-python27/d5a3408/console.html

2014-06-04 18:05:06.828 | FAIL: tests.test_rabbit.TestRacyWaitForReply.test_send_receive
2014-06-04 18:05:06.829 | tags: worker-1
2014-06-04 18:05:06.829 | ----------------------------------------------------------------------
2014-06-04 18:05:06.829 | Empty attachments:
2014-06-04 18:05:06.829 | stdout
2014-06-04 18:05:06.829 |
2014-06-04 18:05:06.829 | stderr: {{{
2014-06-04 18:05:06.829 | Exception in thread Thread-8:
2014-06-04 18:05:06.829 | Traceback (most recent call last):
2014-06-04 18:05:06.829 | File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
2014-06-04 18:05:06.829 | self.run()
2014-06-04 18:05:06.829 | File "/usr/lib/python2.7/threading.py", line 504, in run
2014-06-04 18:05:06.829 | self.__target(*self.__args, **self.__kwargs)
2014-06-04 18:05:06.830 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/tests/test_rabbit.py", line 289, in send_and_wait_for_reply
2014-06-04 18:05:06.830 | timeout=None))
2014-06-04 18:05:06.830 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/.tox/py27/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-06-04 18:05:06.830 | return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-06-04 18:05:06.830 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/.tox/py27/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send
2014-06-04 18:05:06.830 | result = self._waiter.wait(msg_id, timeout)
2014-06-04 18:05:06.830 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/tests/test_rabbit.py", line 279, in reply_waiter
2014-06-04 18:05:06.830 | wait_conditions.pop().wait()
2014-06-04 18:05:06.830 | IndexError: pop from empty list
2014-06-04 18:05:06.830 | }}}
2014-06-04 18:05:06.830 |
2014-06-04 18:05:06.830 | Traceback (most recent call last):
2014-06-04 18:05:06.831 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/tests/test_rabbit.py", line 327, in test_send_receive
2014-06-04 18:05:06.831 | self.assertEqual(len(replies), len(senders))
2014-06-04 18:05:06.831 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 321, in assertEqual
2014-06-04 18:05:06.831 | self.assertThat(observed, matcher, message)
2014-06-04 18:05:06.831 | File "/home/jenkins/workspace/gate-oslo.messaging-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertThat
2014-06-04 18:05:06.831 | raise mismatch_error
2014-06-04 18:05:06.831 | MismatchError: 1 != 2

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRkFJTFxcOiB0ZXN0cy50ZXN0X3JhYmJpdC5UZXN0UmFjeVdhaXRGb3JSZXBseS50ZXN0X3NlbmRfcmVjZWl2ZVwiIEFORCB0YWdzOlwiY29uc29sZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJjdXN0b20iLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsiZnJvbSI6IjIwMTQtMDUtMjJUMjA6MDM6NDgrMDA6MDAiLCJ0byI6IjIwMTQtMDYtMDVUMjA6MDM6NDgrMDA6MDAiLCJ1c2VyX2ludGVydmFsIjoiMCJ9LCJzdGFtcCI6MTQwMTk5ODczMDQ1N30=

Only 1 hit in 2 weeks, but it's in the check queue. It only showed up on this change though:

https://review.openstack.org/#/c/94972/

[Matt Riedemann (mriedem)]

Wondering if this is the fix: https://review.openstack.org/75838