Ceilometer alarm unit test failures on CI check

[Rohit Jaiswal]

Very recently, two of my reviews (https://review.openstack.org/#/c/178435/ and https://review.openstack.org/#/c/187397/) have failed on the same unit test, below is the console log -

2015-06-02 01:08:52.072 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_str(sqlite)
2015-06-02 01:08:52.072 | tags: worker-0
2015-06-02 01:08:52.072 | ----------------------------------------------------------------------
2015-06-02 01:08:52.072 | Empty attachments:
2015-06-02 01:08:52.073 | stderr
2015-06-02 01:08:52.073 | stdout
2015-06-02 01:08:52.073 |
2015-06-02 01:08:52.073 | Traceback (most recent call last):
2015-06-02 01:08:52.073 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:08:52.073 | return func(*args, **kwargs)
2015-06-02 01:08:52.073 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 761, in test_post_invalid_alarm_input_enabled_str
2015-06-02 01:08:52.074 | resp.json['error_message']['faultstring'])
2015-06-02 01:08:52.074 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-06-02 01:08:52.074 | self.assertThat(observed, matcher, message)
2015-06-02 01:08:52.074 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-06-02 01:08:52.074 | raise mismatch_error
2015-06-02 01:08:52.074 | testtools.matchers._impl.MismatchError: !=:
2015-06-02 01:08:52.074 | reference = "Invalid input for field/attribute enabled. Value: 'bad_enabled'. Wrong type. Expected '<type 'bool'>', got '<type 'str'>'"
2015-06-02 01:08:52.075 | actual = u"Invalid input for field/attribute data. Value: '{'name': 'alarm5', 'enabled': 'bad_enabled', 'state': 'ok', 'threshold_rule': {'comparison_operator': 'gt', 'meter_name': 'ameter', 'threshold': 50.0}, 'type': 'threshold'}'. Value not an unambiguous boolean: bad_enabled"
2015-06-02 01:08:52.075 | ======================================================================
2015-06-02 01:08:52.075 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_str(mongodb)
2015-06-02 01:08:52.075 | tags: worker-0
2015-06-02 01:08:52.075 | ----------------------------------------------------------------------
2015-06-02 01:08:52.075 | Empty attachments:
2015-06-02 01:08:52.076 | stderr
2015-06-02 01:08:52.076 | stdout
2015-06-02 01:08:52.076 |
2015-06-02 01:08:56.690 | traceback-1: {{{
2015-06-02 01:08:56.691 | Traceback (most recent call last):
2015-06-02 01:08:56.691 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:08:56.691 | return func(*args, **kwargs)
2015-06-02 01:08:56.691 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 761, in test_post_invalid_alarm_input_enabled_str
2015-06-02 01:08:56.691 | resp.json['error_message']['faultstring'])
2015-06-02 01:08:56.691 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-06-02 01:08:56.691 | self.assertThat(observed, matcher, message)
2015-06-02 01:08:56.692 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-06-02 01:08:56.692 | raise mismatch_error
2015-06-02 01:08:56.692 | testtools.matchers._impl.MismatchError: !=:
2015-06-02 01:08:56.692 | reference = "Invalid input for field/attribute enabled. Value: 'bad_enabled'. Wrong type. Expected '<type 'bool'>', got '<type 'str'>'"
2015-06-02 01:08:56.692 | actual = u"Invalid input for field/attribute data. Value: '{'name': 'alarm5', 'enabled': 'bad_enabled', 'state': 'ok', 'threshold_rule': {'comparison_operator': 'gt', 'meter_name': 'ameter', 'threshold': 50.0}, 'type': 'threshold'}'. Value not an unambiguous boolean: bad_enabled"
2015-06-02 01:08:56.692 | }}}
2015-06-02 01:08:56.692 | ======================================================================
2015-06-02 01:08:56.693 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_str(hbase)
2015-06-02 01:08:56.693 | tags: worker-0
2015-06-02 01:08:56.693 | ----------------------------------------------------------------------
2015-06-02 01:08:56.693 | Empty attachments:
2015-06-02 01:08:56.693 | stderr
2015-06-02 01:08:56.693 | stdout
2015-06-02 01:08:56.693 |
2015-06-02 01:08:56.694 | traceback-2: {{{
2015-06-02 01:08:56.694 | Traceback (most recent call last):
2015-06-02 01:08:56.694 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:08:56.694 | return func(*args, **kwargs)
2015-06-02 01:08:56.694 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 761, in test_post_invalid_alarm_input_enabled_str
2015-06-02 01:08:56.694 | resp.json['error_message']['faultstring'])
2015-06-02 01:08:56.694 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-06-02 01:08:56.695 | self.assertThat(observed, matcher, message)
2015-06-02 01:08:56.695 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-06-02 01:08:56.695 | raise mismatch_error
2015-06-02 01:08:56.695 | testtools.matchers._impl.MismatchError: !=:
2015-06-02 01:08:56.695 | reference = "Invalid input for field/attribute enabled. Value: 'bad_enabled'. Wrong type. Expected '<type 'bool'>', got '<type 'str'>'"
2015-06-02 01:08:56.695 | actual = u"Invalid input for field/attribute data. Value: '{'name': 'alarm5', 'enabled': 'bad_enabled', 'state': 'ok', 'threshold_rule': {'comparison_operator': 'gt', 'meter_name': 'ameter', 'threshold': 50.0}, 'type': 'threshold'}'. Value not an unambiguous boolean: bad_enabled"
2015-06-02 01:08:56.696 | }}}
2015-06-02 01:08:56.696 | ======================================================================
2015-06-02 01:08:56.696 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_str(db2)
2015-06-02 01:08:56.696 | tags: worker-0
2015-06-02 01:08:56.696 | ----------------------------------------------------------------------
2015-06-02 01:08:56.696 | Empty attachments:
2015-06-02 01:08:56.696 | stderr
2015-06-02 01:08:56.697 | stdout
2015-06-02 01:08:56.697 |
2015-06-02 01:08:56.697 | traceback-3: {{{
2015-06-02 01:08:56.697 | Traceback (most recent call last):
2015-06-02 01:08:56.697 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:08:56.697 | return func(*args, **kwargs)
2015-06-02 01:08:56.697 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 761, in test_post_invalid_alarm_input_enabled_str
2015-06-02 01:08:56.698 | resp.json['error_message']['faultstring'])
2015-06-02 01:08:56.698 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual
2015-06-02 01:08:56.698 | self.assertThat(observed, matcher, message)
2015-06-02 01:08:56.698 | File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
2015-06-02 01:08:56.698 | raise mismatch_error
2015-06-02 01:08:56.698 | testtools.matchers._impl.MismatchError: !=:
2015-06-02 01:08:56.698 | reference = "Invalid input for field/attribute enabled. Value: 'bad_enabled'. Wrong type. Expected '<type 'bool'>', got '<type 'str'>'"
2015-06-02 01:08:56.699 | actual = u"Invalid input for field/attribute data. Value: '{'name': 'alarm5', 'enabled': 'bad_enabled', 'state': 'ok', 'threshold_rule': {'comparison_operator': 'gt', 'meter_name': 'ameter', 'threshold': 50.0}, 'type': 'threshold'}'. Value not an unambiguous boolean: bad_enabled"

===============================================
2015-06-02 01:11:53.642 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_int(sqlite)
2015-06-02 01:11:53.642 | tags: worker-6
2015-06-02 01:11:53.642 | ----------------------------------------------------------------------
2015-06-02 01:11:53.642 | Empty attachments:
2015-06-02 01:11:53.643 | stderr
2015-06-02 01:11:53.643 | stdout
2015-06-02 01:11:53.643 |
2015-06-02 01:11:53.643 | Traceback (most recent call last):
2015-06-02 01:11:53.643 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:11:53.643 | return func(*args, **kwargs)
2015-06-02 01:11:53.643 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 785, in test_post_invalid_alarm_input_enabled_int
2015-06-02 01:11:53.644 | resp.json['error_message']['faultstring'])
2015-06-02 01:11:53.644 | KeyError: 'error_message'
2015-06-02 01:11:53.644 | ======================================================================
2015-06-02 01:11:53.644 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_int(mongodb)
2015-06-02 01:11:53.644 | tags: worker-6
2015-06-02 01:11:53.644 | ----------------------------------------------------------------------
2015-06-02 01:11:53.644 | Empty attachments:
2015-06-02 01:11:53.645 | stderr
2015-06-02 01:11:53.645 | stdout
2015-06-02 01:11:53.645 |
2015-06-02 01:11:53.645 | traceback-1: {{{
2015-06-02 01:11:53.645 | Traceback (most recent call last):
2015-06-02 01:11:53.645 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:11:53.645 | return func(*args, **kwargs)
2015-06-02 01:11:53.646 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 785, in test_post_invalid_alarm_input_enabled_int
2015-06-02 01:11:53.646 | resp.json['error_message']['faultstring'])
2015-06-02 01:11:53.646 | KeyError: 'error_message'
2015-06-02 01:11:53.646 | }}}
2015-06-02 01:11:53.646 | ======================================================================
2015-06-02 01:11:53.646 | FAIL: ceilometer.tests.api.v2.test_alarm_scenarios.TestAlarms.test_post_invalid_alarm_input_enabled_int(hbase)
2015-06-02 01:11:53.647 | tags: worker-6
2015-06-02 01:11:53.647 | ----------------------------------------------------------------------
2015-06-02 01:11:53.647 | Empty attachments:
2015-06-02 01:11:53.647 | stderr
2015-06-02 01:11:53.647 | stdout
2015-06-02 01:11:53.647 |
2015-06-02 01:11:53.647 | traceback-2: {{{
2015-06-02 01:11:53.648 | Traceback (most recent call last):
2015-06-02 01:11:53.648 | File "ceilometer/tests/base.py", line 97, in skip_if_not_implemented
2015-06-02 01:11:53.648 | return func(*args, **kwargs)
2015-06-02 01:11:53.648 | File "ceilometer/tests/api/v2/test_alarm_scenarios.py", line 785, in test_post_invalid_alarm_input_enabled_int
2015-06-02 01:11:53.648 | resp.json['error_message']['faultstring'])
2015-06-02 01:11:53.648 | KeyError: 'error_message'
2015-06-02 01:11:53.648 | }}}

[ZhiQiang Fan (aji-zqfan)]

this report is duplicate of https://bugs.launchpad.net/ceilometer/+bug/1460869

and there already has a fix for it: https://review.openstack.org/187425

[Stéphane Bisinger (kjir)]

WSME 0.7.0 introduced checks and coversions from strings to the intended datatype, so now invalid data, like "bad_enabled" for bools, will throw an error. This is intentional and should be updated in the unit tests which should have valid data unless you want to test error scenarios.
The linked Ceilometer bug and patch set correctly address the issue in the ceilometer unit tests.

[Stéphane Bisinger (kjir)]

By looking at those unit test I can't help but wonder: is it useful to test WSME behaviour in Ceilometer? Shouldn't those kind of tests be in WSME rather that Ceilometer?

[gordon chung (chungg)]

we'll track in other bug.

@Stephane, unfortunately, WSME is what is doing the validation and thus it indirectly influences our behaviour