[test] Some tests may fail on slow environments since they expect exact date representation

[Kirill Zaitsev]

specifically

murano_tempest_tests.tests.api.application_catalog.test_env_templates.TestEnvironmentTemplates.test_get_public_private_both_env_templates[smoke]
murano_tempest_tests.tests.api.application_catalog.test_sessions.TestSessions.test_get_session[smoke]

The error boils down to 2016-09-22T20:13:08 != 2016-09-22T20:13:09

other tests might be affected too. We need to exclude time from tests or instead of comparing exact time representation — compare timedelta with a certain allowance

Here are full tracebacks:

murano_tempest_tests.tests.api.application_catalog.test_env_templates.TestEnvironmentTemplates.test_get_public_private_both_env_templates[smoke]
------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/plugins/murano/murano_tempest_tests/tests/api/application_catalog/test_env_templates.py", line 163, in test_get_public_private_both_env_templates
        self.assertIn(public_env_template, public_env_templates)
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
        self.assertThat(haystack, Contains(needle), message)
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: {u'updated': u'2016-09-22T20:13:08', u'name': u'get_public_private_both_bb2935de', u'created': u'2016-09-22T20:13:08', u'tenant_id': u'b7a1ef2908d34bb2ad9060de5ddb3370', u'version': 0, u'description_text': u'', u'is_public': True, u'id': u'00a1eb35b62b418a99bf349184a8fbbf'} not in [{u'updated': u'2016-09-22T20:13:09', u'name': u'get_public_private_both_bb2935de', u'created': u'2016-09-22T20:13:09', u'tenant_id': u'b7a1ef2908d34bb2ad9060de5ddb3370', u'version': 0, u'description_text': u'', u'is_public': True, u'id': u'00a1eb35b62b418a99bf349184a8fbbf'}, {u'updated': u'2016-09-22T20:13:07', u'name': u'TestEnvironmentTemplates_03112244', u'created': u'2016-09-22T20:12:55', u'tenant_id': u'b7a1ef2908d34bb2ad9060de5ddb3370', u'version': 0, u'description_text': u'', u'is_public': True, u'id': u'6c00f7530e5d440c9f0b5835a42164f4'}]

and

murano_tempest_tests.tests.api.application_catalog.test_sessions.TestSessions.test_get_session[smoke]
-----------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/plugins/murano/murano_tempest_tests/tests/api/application_catalog/test_sessions.py", line 59, in test_get_session
        self.assertEqual(session, session_from_resp)
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/rally/.rally/tempest/for-deployment-44cea41c-ad13-4fd8-9b99-fc00b7cea2e1/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: !=:
    reference = {u'created': u'2016-09-22T20:12:43',
     u'environment_id': u'661ff5e27fe74226bb1d1fc1c8165fd0',
     u'id': u'cb6de9e38ab3459286cb834703c5e482',
     u'state': u'opened',
     u'updated': u'2016-09-22T20:12:43',
     u'user_id': u'b6eaeaadbab4427c9e9a10ca2e01b584',
     u'version': 0}
    actual = {u'created': u'2016-09-22T20:12:44',
     u'environment_id': u'661ff5e27fe74226bb1d1fc1c8165fd0',
     u'id': u'cb6de9e38ab3459286cb834703c5e482',
     u'state': u'opened',
     u'updated': u'2016-09-22T20:12:44',
     u'user_id': u'b6eaeaadbab4427c9e9a10ca2e01b584',
     u'version': 0}

[Victor Ryzhenkin (vryzhenkin)]

also client affected

muranoclient.tests.functional.cli.test_murano.EnvTemplateMuranoSanityClientTest.test_environment_template_create

[Kirill Zaitsev (kzaitsev)]

muranoclient.tests.functional.cli.test_murano.EnvTemplateMuranoSanityClientTest.test_environment_template_create
----------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "muranoclient/tests/functional/cli/test_murano.py", line 360, in test_environment_template_create
        self.assertIn(env_template, env_template_list)
      File "/root/python-muranoclient/.tox/functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
        self.assertThat(haystack, Contains(needle), message)
      File "/root/python-muranoclient/.tox/functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: {'Updated': '2016-09-22T21:10:06', 'Name': 'TestMuranoSanityEnvTemp_3909e0e6', 'Created': '2016-09-22T21:10:06', 'Is public': '', 'ID': 'f46369f5d80347c3afabe6f3e0428d15'} not in [{'Updated': '2016-09-22T20:43:50', 'Name': 'MuranoTestTS-env-tmp-create_87e47868', 'Created': '2016-09-22T20:43:50', 'Is public': '', 'ID': '05e0c159a9b54a3787f1c1d47ec871b0'}, {'Updated': '2016-09-22T20:36:39', 'Name': 'TestMuranoSanityEnvTemp_2c3b6770', 'Created': '2016-09-22T20:36:39', 'Is public': '', 'ID': '13e85e7e1649464f8207a223ab7d15a5'}, {'Updated': '2016-09-22T20:41:54', 'Name': 'TestMuranoSanityEnvTemp_9a89e97c', 'Created': '2016-09-22T20:41:54', 'Is public': '', 'ID': '324a3ecdc24046789032f55925f37976'}, {'Updated': '2016-09-22T20:41:26', 'Name': 'TestMuranoSanityEnvTemp_b880389d', 'Created': '2016-09-22T20:41:26', 'Is public': '', 'ID': '7d95134ae5844199b5cda101baa6b230'}, {'Updated': '2016-09-22T20:41:48', 'Name': 'TestMuranoSanityEnvTemp_f6093813', 'Created': '2016-09-22T20:41:48', 'Is public': '', 'ID': 'a703007018dd4703a022bdb8cd53c07b'}, {'Updated': '2016-09-22T20:36:34', 'Name': 'TestMuranoSanityEnvTemp_cd2829c6', 'Created': '2016-09-22T20:36:34', 'Is public': '', 'ID': 'b304169b28bc4c438930bcc8cd8cd4f1'}, {'Updated': '2016-09-22T20:41:39', 'Name': 'TestMuranoSanityEnvTemp_dde2c67a', 'Created': '2016-09-22T20:41:39', 'Is public': 'True', 'ID': 'b3bfe0335ca143a98a631f6d2b013bee'}, {'Updated': '2016-09-22T21:05:09', 'Name': 'TestMuranoSanityEnvTemp_7b6092e6', 'Created': '2016-09-22T21:05:09', 'Is public': '', 'ID': 'da378d775ee8429bacc5a4cc4c1b2241'}, {'Updated': '2016-09-22T20:36:22', 'Name': 'TestMuranoSanityEnvTemp_29e6b5e2', 'Created': '2016-09-22T20:36:22', 'Is public': 'True', 'ID': 'e1fa263687714df8a803545df329b82f'}, {'Updated': '2016-09-22T20:41:31', 'Name': 'TestMuranoSanityEnvTemp_2e652a28', 'Created': '2016-09-22T20:41:31', 'Is public': '', 'ID': 'e7e5eef0d9ce4026a31d784e3c4ff179'}, {'Updated': '2016-09-22T21:10:07', 'Name': 'TestMuranoSanityEnvTemp_3909e0e6', 'Created': '2016-09-22T21:10:07', 'Is public': '', 'ID': 'f46369f5d80347c3afabe6f3e0428d15'}, {'Updated': '2016-09-22T20:36:28', 'Name': 'TestMuranoSanityEnvTemp_d2ea5939', 'Created': '2016-09-22T20:36:28', 'Is public': '', 'ID': 'f5f15899e608411d8618c349ec484fa1'}]

[Ilya Popov (ilya-p)]

Kirill,

I'm not sure the way how the time could be different for muranoclient.tests.functional.cli.test_murano.EnvTemplateMuranoSanityClientTest.test_environment_template_create:

if

       env_template = self.create_murano_object('env-template','TestMuranoSanityEnvTemp')

finished successfully, it was add db entity.

And
       env_template_list = self.listing('env-template-list')

should return the same dates/times

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/396346

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396346
Committed: https://git.openstack.org/cgit/openstack/python-muranoclient/commit/?id=5260570c953ef8be61ccd3f1ee908f151d64e801
Submitter: Jenkins
Branch: master

commit 5260570c953ef8be61ccd3f1ee908f151d64e801
Author: Ilya Popov <email address hidden>
Date: Thu Nov 10 21:09:06 2016 +0300

    Fix removes date_time items from dictionaries

    Some functional tests fail because they assert dictionaries
    with date/time items. For example, test_environment_template_create
    checks that dictionary env_template exists in env_template_list
    dicrionary list. All mentioned dictionaries have date/time items
    with keys 'Created' and 'Updated'. If the system, where functional
    tests are executed will be quite slow - time in these dictionaries
    could differ. This behavior will result corresponding test to fail
    even if all tasks will be executed successfully because time in
    dictionaries for 'Created' and 'Updated' keys will differ and
    assert will fail

    Removing the date/time items from dictionaries will help to
    successfully pass functional tests

    Closes-Bug: #1626733

    Change-Id: I859eb7024619c8718411e295fe7cb93c6b8972eb

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/396891

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396891
Committed: https://git.openstack.org/cgit/openstack/python-muranoclient/commit/?id=8007de89b1316e04fafce58d10630974b3523b60
Submitter: Jenkins
Branch: master

commit 8007de89b1316e04fafce58d10630974b3523b60
Author: Ilya Popov <email address hidden>
Date: Sun Nov 13 01:13:13 2016 +0300

    Fix removes date_time items from dictionaries

    Some functional tests fail because they assert dictionaries
    with date/time items. For example, test_environment_template_create
    checks that dictionary env_template exists in env_template_list
    dicrionary list. All mentioned dictionaries have date/time items
    with keys 'Created' and 'Updated'. If the system, where functional
    tests are executed will be quite slow - time in these dictionaries
    could differ. This behavior will result corresponding test to fail
    even if all tasks will be executed successfully because time in
    dictionaries for 'Created' and 'Updated' keys will differ and
    assert will fail

    Removing the date/time items from dictionaries will help to
    successfully pass functional tests

    Related-Bug: #1626733

    Change-Id: Ibb85992e8b3869ae2b7adf58f8b4f9b134d07139

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/404035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/404035
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=787f14baa273486c44e88f2dc5e6e0b9cae2a35e
Submitter: Jenkins
Branch: master

commit 787f14baa273486c44e88f2dc5e6e0b9cae2a35e
Author: Ilya Popov <email address hidden>
Date: Tue Nov 29 09:30:11 2016 +0300

    Fix removes date_time items from dictionaries

    Some functional tests fail because they assert dictionaries
    with date/time items. For example, test_environment_template_create
    checks that dictionary env_template exists in env_template_list
    dicrionary list. All mentioned dictionaries have date/time items
    with keys 'Created' and 'Updated'. If the system, where functional
    tests are executed will be quite slow - time in these dictionaries
    could differ. This behavior will result corresponding test to fail
    even if all tasks will be executed successfully because time in
    dictionaries for 'Created' and 'Updated' keys will differ and
    assert will fail

    Removing the date/time items from dictionaries will help to
    successfully pass functional tests

    Closes-Bug: #1626733

    Change-Id: Ife3e157d9c3e8f0a6c1680b6134fd2c40dd62faf

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 3.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-muranoclient 0.12.0 release.