OSTF: Heat test Check creation of stack with Wait Condition/Handle resources failed by timeout on step wait stack CREATE_COMPLETE status

[Tatyanka]

Steps:
Deploy:
3 controllers+ceph
2 computes

run ostf

All test failed except:
 'Check creation of stack with Wait Condition/Handle resources' (622.6 s) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

single run of platform suit finish fail too:
root@nailgun ~]# fuel health --env 1 --check tests_platform
[ 1 of 5] [success] 'Typical stack actions: create, delete, show details, etc.' (113.5 s)
[ 2 of 5] [success] 'Advanced stack actions: suspend, resume and check' (129.9 s)
[ 3 of 5] [success] 'Check stack rollback' (161.4 s)
[ 4 of 5] [success] 'Update stack actions: inplace, replace and update whole template' (146.2 s)
[ 5 of 5] [failure] 'Check creation of stack with Wait Condition/Handle resources' (622.6 s) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/278534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278534
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=d96b55d316048b97294bd49d46f268b426f9e06b
Submitter: Jenkins
Branch: master

commit d96b55d316048b97294bd49d46f268b426f9e06b
Author: Tatyana Leontovich <email address hidden>
Date: Wed Feb 10 19:27:13 2016 +0200

    Temporary disable platfrom test execution

    Change-Id: I33f4ed574325837972739bf8e321353dbb0f2a4d
    Related-Bug: #1544179

[Timur Nurlygayanov (tnurlygayanov)]

The fix was merged, I've changed the status to Fix Commited.

[Artem Hrechanychenko (agrechanichenko)]

Reproduced on CI:https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.network_templates/33/testReport/%28root%29/deploy_cinder_net_tmpl/deploy_cinder_net_tmpl/

raceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/tests/test_net_templates.py", line 108, in deploy_cinder_net_tmpl
    'ha', 'tests_platform'])
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/models/fuel_web_client.py", line 1077, in run_ostf
    failed_test_name=failed_test_name, test_sets=test_sets)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/models/fuel_web_client.py", line 273, in assert_ostf_run
    indent=1)))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check creation of stack with Wait Condition/Handle resources (failure) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

[Dmitry Kabanov (dkabanov)]

I am sure that original reason of the failure was incorrect configuration of Heat.
We are using IPs from Management network in "heat_waitcondition_server_url", "heat_watch_server_url"
and "heat_metadata_server_url" variables. But these URLs should be accessable from VMs booted in the cloud.
In fuel-library (from 7.0 version) we are providing "management_vip" as an "external_ip" to 'openstack::heat'.
The "heat_waitcondition_server_url", "heat_watch_server_url" and "heat_metadata_server_url" variables will be
configured in 'openstack::heat' with using "external_ip" variable if these variables are not provided.

fuel-library/deployment/puppet/osnailyfacter/modular/heat/heat.pp:

class { 'openstack::heat' :
  external_ip => $management_vip,
  ...
  (We do not have "heat_waitcondition_server_url", "heat_watch_server_url" and "heat_metadata_server_url" here)
  ...
}

fuel-library/deployment/puppet/openstack/manifests/heat.pp:

class openstack::heat (
  ...
  $heat_metadata_server_url = false,
  $heat_waitcondition_server_url = false,
  $heat_watch_server_url = false,
  ...
)
  if $heat_waitcondition_server_url {
    $waitcondition_server_url = $heat_waitcondition_server_url
  } else {
    $waitcondition_server_url = "${heat_protocol}://${external_ip}:${api_cfn_bind_port}/v1/waitcondition"
  }

 ("heat_watch_server_url" and "heat_metadata_server_url" variables will be configured here too.)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287229
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=aa8375acf44f730df5de1df4202ada2c7a7be4f2
Submitter: Jenkins
Branch: master

commit aa8375acf44f730df5de1df4202ada2c7a7be4f2
Author: Alexandr Nagovitsyn <email address hidden>
Date: Wed Mar 2 17:32:16 2016 +0300

    Add RAM filters and increased the timers in heat tests

    Recently, a few new bugs associated with timers and error
    'failed by timeout on step wait stack CREATE_COMPLETE status'
    Research has shown that an error in most cases occurs due to a
    slow environments.
    Typically, a small amount of RAM (<3 GB).
    Tests autoscaling and wait condition require more than 7 GB
    Also increased the other timers in (+ 120s in verify steps,
    + 30s in create stack steps, +10s in some other.

    Change-Id: I8066581e6358e84655ac3b419dcaa324e4f57aca
    Closes-bug:#1544179
    Closes-bug:#1539721
    Closes-bug:#1525200

[Alexander Nagovitsyn (gluk12189)]

Done

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285595
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=eb22a9246287c59f40d0f51f26afd24a81b0e66f
Submitter: Jenkins
Branch: master

commit eb22a9246287c59f40d0f51f26afd24a81b0e66f
Author: Andrew Woodward <email address hidden>
Date: Fri Feb 26 16:53:45 2016 -0800

    Drop openstack::heat

    This uplifts the openstack::heat directly into the calling task.

    Closes-bug: #1544179
    Change-Id: Id3a39e0c75832e64c2f60664c0ffc0acfa7c0ec3
    Blueprint: fuel-remove-conflict-openstack

[Andrew Woodward (xarses)]

Please confirm if #285595 resolves the problem in CI

[Timur Nurlygayanov (tnurlygayanov)]

Ok, the issue was fixed, marked as Fix Released.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.

[Alexey Stupnikov (astupnikov)]

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.