[OSTF] "Check stack autoscaling and Typical stack actions" tests fails with "Time limit exceeded"

[Artem Hrechanychenko]

Check stack autoscaling and Typical stack actions" tests fails with "Time limit exceeded"

"Typical stack actions: create, delete, show details, etc.
Time limit exceeded while waiting for stack creation to finish. Please refer to OpenStack logs for more details."

"Check stack autoscaling
Time limit exceeded while waiting for launching the new instance per autoscaling alarm to finish. Please refer to OpenStack logs for more details."

Steps to reproduce:

1)Create env with Neutron VLAN, Ubuntu, Ceilometer
2) Add 1 controller, 1 compute, 1 cinder + mongo, 2 mongo
3) Deploy Cluster
4) Verify network
5) Run OSTF

Actual Result:

"Typical stack actions: create, delete, show details, etc.
Time limit exceeded while waiting for stack creation to finish. Please refer to OpenStack logs for more details."
(nose_storage_plugin) fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_actions Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 340, in run testMethod() File "/usr/lib/python2.6/site-packages/fuel_health/tests/tests_platform/test_heat.py", line 247, in test_actions template, parameters=parameters File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 183, in verify " Please refer to OpenStack logs for more details.") File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail raise self.failureException(msg) AssertionError: Step 2 failed: Time limit exceeded while waiting for stack creation to finish. Please refer to OpenStack logs for more details.

"Check stack autoscaling
Time limit exceeded while waiting for launching the new instance per autoscaling alarm to finish. Please refer to OpenStack logs for more details."
(nose_storage_plugin) fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_autoscaling Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 340, in run testMethod() File "/usr/lib/python2.6/site-packages/fuel_health/tests/tests_platform/test_heat.py", line 711, in test_autoscaling len(instances) + 1, 480, 10, reduced_stack_name File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 183, in verify " Please refer to OpenStack logs for more details.") File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail raise self.failureException(msg) AssertionError: Step 11 failed: Time limit exceeded while waiting for launching the new instance per autoscaling alarm to finish. Please refer to OpenStack logs for more details

{"build_id": "2015-08-14_17-24-26", "build_number": "176", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-14_17-24-26", "build_number": "176", "api": "1.0", "fuel-library_sha": "db1c04c9de65c8e107475967d86c7a5791ef3853", "nailgun_sha": "eaaf1233c613c6e08bcf99ee09ebb1bdb7dd4e31", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863", "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d", "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d", "release": "7.0", "fuelmain_sha": "d8c726645be087bc67e2eeca134f0f9747cfeacd"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "db1c04c9de65c8e107475967d86c7a5791ef3853", "nailgun_sha": "eaaf1233c613c6e08bcf99ee09ebb1bdb7dd4e31", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863", "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d", "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d", "release": "7.0", "fuelmain_sha": "d8c726645be087bc67e2eeca134f0f9747cfeacd"}

[Timur Nurlygayanov (tnurlygayanov)]

Hi Anastasia,

please, take a look.

[Artem Hrechanychenko (agrechanichenko)]

I deployed cluster of a few times with different Acceptance cases and this issue still present

KVM+Ubuntu+NeutronVlan+Ceilometer+Untagged management VLAN

KVM+Ubuntu+NeutronVlan+Ceilometer

[Dmitry Borodaenko (angdraug)]

"some (sic!) test times out" is an unacceptable summary for a bug, if we leave it like that every time anything related to ostf reports a timeout this bug will be reopened. Please make it more specific asap.

[Anastasia Kuznetsova (akuznetsova)]

There are no errors /traces in heat/ceilometer logs which would point to any problems and lead to timeout errors. Problem also is not in cluster configuration.
My point is that:
1) 176 ISO is not stable by itself and maybe there are some problems with rabbit/connectivity/etc which can cause timeout issues:
--- this ISO was not tested on BVT jobs
--- results of SWARM job for this ISO is unstable
(here all tests were failed:
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha_one_controller/60/
and here all is ok:
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.services_ha_one_controller/61/)
2) FUEL environment is too slow (there is 2 Gb of RAM and 1 CPU on compute node), for example first test "Typical stack actions" failed because after stack was created test couldn't get/receive stack info during 20 sec (it is large timeout for such operation)

We can't increase timeout infinitely every time when tests fails in some slow/unstable environments.