Fuel 4.0 ostf murano sanity check fails: "Time limit exceeded while waiting for creating environment to finish."

[Aleksandr Didenko]

ISO: {"release": "4.0", "nailgun_sha": "1c25678d80bc6d3940abf897ac914662c6c8af7e", "ostf_sha": "18830ef63c9499cd7244a811717aa20d5c4aa58e", "astute_sha": "83a8115245e3314d770a4e858317713ac0dbee48", "fuellib_sha": "2d9ae05f9a08e51bdabdca9baaab6e6941e8cb79"}

OSTF test result:
[sanity] Create and delete Murano environment: failure
        Time limit exceeded while waiting for creating environment to finish. Please refer to OpenStack logs for more details.

It happened only once during the first OSTF run right after OpenStack env deployment. After this first OSTF run there was "ost1_test-Murano_env01" environment with "Ready to configure" status in "Murano" Horizon tab. So it looks like OSTF was able to create murano test env but it took more time than configured in the check timeout.

I've then run several OSTF tests and was not able to reproduce this issue.

Part of /var/log/ostf-stdout.log:

2013-12-18 09:10:04 ERROR (nose_storage_plugin) fuel_health.tests.sanity.test_sanity_murano.MuranoSanityTests.test_create_and_delete_service
Traceback (most recent call last):
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/unittest2/case.py", line 340, in run
    testMethod()
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/tests/sanity/test_sanity_murano.py", line 43, in test_create_and_delete_service
    "ost1_test-Murano_env01")
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 186, in verify
    " Please refer to OpenStack logs for more details.")
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail
    raise self.failureException(msg)
AssertionError: Step 1 failed: Time limit exceeded while waiting for creating environment to finish. Please refer to OpenStack logs for more details.
-------------------- >> begin captured logging << --------------------
fuel_health.config: INFO: INSTANCE {20610: <fuel_health.config.NailgunConfig object at 0x2a4a110>}
fuel_health.config: INFO: INSTANCE {20610: <fuel_health.config.NailgunConfig object at 0x2a4a110>}
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
fuel_health.nmanager: DEBUG: Savana url is http://172.16.0.2:8386/v1.0
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "GET http://192.168.0.2:35357/v2.0/tenants HTTP/1.1" 200 276
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "GET http://192.168.0.2:35357/v2.0/tenants HTTP/1.1" 200 276
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 192.168.50.6
requests.packages.urllib3.connectionpool: DEBUG: "POST http://172.16.0.2:5000/v2.0/tokens HTTP/1.1" 200 3343
fuel_health.nmanager: WARNING: Can not initialize ceilometer client
muranoclient.common.http: DEBUG: curl -i -X POST -H 'X-Auth-Token: 0287543ca9704fa0b4f3d3b6a900a698' -H 'Content-Type: application/json' -H 'User-Agent: python-muranoclient' -d '{"name": "ost1_test-Murano_env01"}' http://172.16.0.2:8082/environments
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 180, in verify
    result = func(*args, **kwargs)
  File "/opt/fuel_plugins/ostf/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 223, in __exit__
    raise AssertionError(msg)
AssertionError: Time limit exceeded while waiting for creating environment to finish.

--------------------- >> end captured logging << ---------------------

Diagnostic snapshot attached.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/63366

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/63366
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=aee4582c2f5c5e3e7b8f0ac370d32e190d34f18d
Submitter: Jenkins
Branch: master

commit aee4582c2f5c5e3e7b8f0ac370d32e190d34f18d
Author: TimurNurlygayanov <email address hidden>
Date: Fri Dec 20 13:21:15 2013 +0400

    Fixed timeouts for Murano sanity test

    Closes-bug: #1262134

    Change-Id: I5aec632facc5e9de482cd931ecc092592879b850