oslo.messaging 2.6.1 breaks ceilometer unit tests

Bug #1505730 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
Unassigned
oslo.messaging
Fix Released
Critical
Joshua Harlow

Bug Description

When running the ceilometer units tests with olso.messaging 2.61 some tests will timeout as follows. Downgrading to 2.5.0 fixes the issue (2.6.0 has unrelated problems with ZMQ constants). More details to follow as I have them.

======================================================================
FAIL: ceilometer.tests.unit.publisher.test_messaging_publisher.RpcOnlyPublisherTest.test_published_no_mock
tags: worker-2
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "ceilometer/tests/unit/publisher/test_messaging_publisher.py", line 122, in test_published_no_mock
    collector.wait()
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/oslo_messaging/server.py", line 177, in wait
    self._executor.wait()
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/oslo_messaging/_executors/impl_pooledexecutor.py", line 122, in wait
    self._tombstone.wait()
  File "/usr/lib64/python2.7/threading.py", line 624, in wait
    self.__cond.wait(timeout, balancing)
  File "/usr/lib64/python2.7/threading.py", line 340, in wait
    waiter.acquire()
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/eventlet/semaphore.py", line 113, in acquire
    hubs.get_hub().switch()
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
    self.wait(sleep_time)
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 82, in wait
    sleep(seconds)
  File "/home/cdent/src/ceilometer/.tox/py27/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException

Chris Dent (cdent)
description: updated
Revision history for this message
Chris Dent (cdent) wrote :

In ceilo there are three timing-out tests:

ceilometer.tests.unit.alarm.test_rpc.TestRPCAlarmNotifier.test_notify_alarm
ceilometer.tests.unit.alarm.test_rpc.TestRPCAlarmNotifier.test_notify_non_string_reason
ceilometer.tests.unit.publisher.test_messaging_publisher.RpcOnlyPublisherTest.test_published_no_mock

Revision history for this message
Chris Dent (cdent) wrote :

Messaging fix: https://review.openstack.org/#/c/234332/

(this corrects the problems in the three ceilo tests above)

Changed in oslo.messaging:
assignee: nobody → Davanum Srinivas (DIMS) (dims-v)
status: New → In Progress
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Changed in oslo.messaging:
importance: Undecided → Critical
assignee: Davanum Srinivas (DIMS) (dims-v) → Joshua Harlow (harlowja)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (master)

Reviewed: https://review.openstack.org/234372
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=c68266b36b5d8e94fef2cca23ea459a01e05a0d2
Submitter: Jenkins
Branch: master

commit c68266b36b5d8e94fef2cca23ea459a01e05a0d2
Author: Mehdi Abaakouk <email address hidden>
Date: Tue Oct 13 18:30:50 2015 +0200

    Use a condition (and/or a dummy one) instead of a lock

    Instead of having to spin in the wait method, just use
    a condition and block until stopping has actually happened,
    when stop happens, it will use the notify_all method to let
    any blockers release.

    Closes-Bug: #1505730

    Change-Id: I3cfbe1bf02d451e379b1dcc23dacb0139c03be76

Changed in oslo.messaging:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/235049
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=bbcfcbb1af01c64b6f61ce3d15c5303dd6be26aa
Submitter: Jenkins
Branch: master

commit bbcfcbb1af01c64b6f61ce3d15c5303dd6be26aa
Author: liusheng <email address hidden>
Date: Thu Oct 15 09:20:00 2015 +0800

    Block oslo.messaging 2.6.1 release

    The oslo.messaging-2.6.1 has broken Ceilometer Jenkins job two days, The
    change[1] could hopefully fix the issue, but it is better to block the
    oslo.messaging-2.6.1 to make Ceilometer Jenkins alive now.

    [1] https://review.openstack.org/#/c/234372/

    Related-Bug: #1505730
    Change-Id: Ibb2a8c18651c601c844c3bc9d89f18ae032accf8
    Depends-On: Id1e58a9db0d277de2f30de54db97040811f40ff2

Revision history for this message
gordon chung (chungg) wrote :
Changed in ceilometer:
status: New → Fix Committed
Changed in oslo.messaging:
milestone: none → 2.7.0
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Liusheng (liusheng)
Changed in ceilometer:
milestone: none → mitaka-1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.