Time after time get failure on Ubuntu Trusty (failure harder to reproduce on Fedora or MacOS)
For Ubuntu guaranteed way to reproduce is a command like this:
for i in {1..10}; do tox -epy27 'oslo_messaging.tests.drivers.zmq.test_zmq_ack_manager.TestZmqAckManager'; done
about 3 times out of 10 is a failure.
The trace for the failure is:
FAIL: oslo_messaging.tests.drivers.zmq.test_zmq_ack_manager.TestZmqAckManager.test_cast_failure_exhausted_retries
tags: worker-1
----------------------------------------------------------------------
Empty attachments:
stdout
stderr: {{{
WARNING [oslo_messaging._drivers.zmq_driver.client.zmq_ack_manager] No ack received within 5 seconds for 800f4210-6c51-11e6-b38d-40167e709c0f
WARNING [oslo_messaging._drivers.zmq_driver.client.zmq_ack_manager] No ack received within 5 seconds for 800f4210-6c51-11e6-b38d-40167e709c0f
WARNING [oslo_messaging._drivers.zmq_driver.server.consumers.zmq_dealer_consumer] [ozamiatin-pc/topic.server/fef18ec4-7978-457d-a23b-28604aefc92b] Dropping duplicate CAST message 800f4210-6c51-11e6-b38d-40167e709c0f
WARNING [oslo_messaging._drivers.zmq_driver.client.zmq_ack_manager] No ack received within 5 seconds for 800f4210-6c51-11e6-b38d-40167e709c0f
WARNING [oslo_messaging._drivers.zmq_driver.client.zmq_ack_manager] Exhausted number of retries for 800f4210-6c51-11e6-b38d-40167e709c0f
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
ERROR [oslo_messaging._drivers.zmq_driver.proxy.zmq_queue_proxy] Received message with wrong format
}}}
Traceback (most recent call last):
File "/home/admin/zmq-acks/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "oslo_messaging/tests/drivers/zmq/test_zmq_ack_manager.py", line 183, in test_cast_failure_exhausted_retries
self.assertEqual(3, lost_ack_mock.call_count)
File "/home/admin/zmq-acks/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/home/admin/zmq-acks/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 3 != 2
Ran 6 tests in 20.023s (+0.005s)
FAILED (id=15, failures=1 (+1))
error: testr failed (1)
FAIL: oslo_messaging. tests.drivers. zmq.test_ zmq_ack_ manager. TestZmqAckManag er.test_ cast_failure_ exhausted_ retries ------- ------- ------- ------- ------- ------- ------- ------- -------
tags: worker-1
-------
Empty attachments:
stdout
stderr: {{{ ._drivers. zmq_driver. client. zmq_ack_ manager] No ack received within 5 seconds for a417aaae- 6c5f-11e6- b2fe-40167e709c 0f ._drivers. zmq_driver. client. zmq_ack_ manager] No ack received within 5 seconds for a417aaae- 6c5f-11e6- b2fe-40167e709c 0f ._drivers. zmq_driver. server. consumers. zmq_dealer_ consumer] [ozamiatin- pc/topic. server/ 7dc38f34- 92c1-485f- b2fe-25452a1219 18] Dropping duplicate CAST message a417aaae- 6c5f-11e6- b2fe-40167e709c 0f ._drivers. zmq_driver. client. zmq_ack_ manager] No ack received within 5 seconds for a417aaae- 6c5f-11e6- b2fe-40167e709c 0f ._drivers. zmq_driver. client. zmq_ack_ manager] Exhausted number of retries for a417aaae- 6c5f-11e6- b2fe-40167e709c 0f
WARNING [oslo_messaging
WARNING [oslo_messaging
WARNING [oslo_messaging
WARNING [oslo_messaging
WARNING [oslo_messaging
}}}
Traceback (most recent call last): admin/zmq- acks/oslo. messaging/ .tox/py27/ local/lib/ python2. 7/site- packages/ mock/mock. py", line 1305, in patched /tests/ drivers/ zmq/test_ zmq_ack_ manager. py", line 183, in test_cast_ failure_ exhausted_ retries assertEqual( 3, lost_ack_ mock.call_ count) admin/zmq- acks/oslo. messaging/ .tox/py27/ local/lib/ python2. 7/site- packages/ testtools/ testcase. py", line 411, in assertEqual assertThat( observed, matcher, message) admin/zmq- acks/oslo. messaging/ .tox/py27/ local/lib/ python2. 7/site- packages/ testtools/ testcase. py", line 498, in assertThat matchers. _impl.MismatchE rror: 3 != 2
File "/home/
return func(*args, **keywargs)
File "oslo_messaging
self.
File "/home/
self.
File "/home/
raise mismatch_error
testtools.
Ran 6 tests in 20.016s (+0.001s)
FAILED (id=9, failures=1 (+1))
error: testr failed (1)