OSTF: Heat test Check creation of stack with Wait Condition/Handle resources failed by timeout on step wait stack CREATE_COMPLETE status

Bug #1544179 reported by Tatyanka on 2016-02-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
MOS QA Team
6.1.x
High
MOS QA Team
7.0.x
High
MOS QA Team
8.0.x
High
MOS QA Team
Mitaka
High
MOS QA Team

Bug Description

Steps:
Deploy:
3 controllers+ceph
2 computes

run ostf

All test failed except:
 'Check creation of stack with Wait Condition/Handle resources' (622.6 s) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

single run of platform suit finish fail too:
root@nailgun ~]# fuel health --env 1 --check tests_platform
[ 1 of 5] [success] 'Typical stack actions: create, delete, show details, etc.' (113.5 s)
[ 2 of 5] [success] 'Advanced stack actions: suspend, resume and check' (129.9 s)
[ 3 of 5] [success] 'Check stack rollback' (161.4 s)
[ 4 of 5] [success] 'Update stack actions: inplace, replace and update whole template' (146.2 s)
[ 5 of 5] [failure] 'Check creation of stack with Wait Condition/Handle resources' (622.6 s) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

Tatyanka (tatyana-leontovich) wrote :
tags: added: module-ostf
Ilya Kutukov (ikutukov) on 2016-02-10
tags: added: area-qa
Changed in fuel:
status: New → Won't Fix
status: Won't Fix → Confirmed

Reviewed: https://review.openstack.org/278534
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=d96b55d316048b97294bd49d46f268b426f9e06b
Submitter: Jenkins
Branch: master

commit d96b55d316048b97294bd49d46f268b426f9e06b
Author: Tatyana Leontovich <email address hidden>
Date: Wed Feb 10 19:27:13 2016 +0200

    Temporary disable platfrom test execution

    Change-Id: I33f4ed574325837972739bf8e321353dbb0f2a4d
    Related-Bug: #1544179

tags: added: swarm-blocker
Changed in fuel:
assignee: MOS QA Team (mos-qa) → Alexander Nagovitsyn (gluk12189)
Changed in fuel:
status: Confirmed → In Progress

The fix was merged, I've changed the status to Fix Commited.

Changed in fuel:
status: In Progress → Fix Committed

Reproduced on CI:https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.network_templates/33/testReport/%28root%29/deploy_cinder_net_tmpl/deploy_cinder_net_tmpl/

raceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/tests/test_net_templates.py", line 108, in deploy_cinder_net_tmpl
    'ha', 'tests_platform'])
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/models/fuel_web_client.py", line 1077, in run_ostf
    failed_test_name=failed_test_name, test_sets=test_sets)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/models/fuel_web_client.py", line 273, in assert_ostf_run
    indent=1)))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check creation of stack with Wait Condition/Handle resources (failure) Time limit exceeded while waiting for stack status becoming "CREATE_COMPLETE" to finish. Please refer to OpenStack logs for more details.

Changed in fuel:
status: Fix Committed → Confirmed
Changed in fuel:
status: Confirmed → In Progress
tags: added: area-ostf
removed: area-qa module-ostf
Dmitry Kabanov (dkabanov) wrote :

I am sure that original reason of the failure was incorrect configuration of Heat.
We are using IPs from Management network in "heat_waitcondition_server_url", "heat_watch_server_url"
and "heat_metadata_server_url" variables. But these URLs should be accessable from VMs booted in the cloud.
In fuel-library (from 7.0 version) we are providing "management_vip" as an "external_ip" to 'openstack::heat'.
The "heat_waitcondition_server_url", "heat_watch_server_url" and "heat_metadata_server_url" variables will be
configured in 'openstack::heat' with using "external_ip" variable if these variables are not provided.

fuel-library/deployment/puppet/osnailyfacter/modular/heat/heat.pp:

class { 'openstack::heat' :
  external_ip => $management_vip,
  ...
  (We do not have "heat_waitcondition_server_url", "heat_watch_server_url" and "heat_metadata_server_url" here)
  ...
}

fuel-library/deployment/puppet/openstack/manifests/heat.pp:

class openstack::heat (
  ...
  $heat_metadata_server_url = false,
  $heat_waitcondition_server_url = false,
  $heat_watch_server_url = false,
  ...
)
  if $heat_waitcondition_server_url {
    $waitcondition_server_url = $heat_waitcondition_server_url
  } else {
    $waitcondition_server_url = "${heat_protocol}://${external_ip}:${api_cfn_bind_port}/v1/waitcondition"
  }

 ("heat_watch_server_url" and "heat_metadata_server_url" variables will be configured here too.)

Reviewed: https://review.openstack.org/287229
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=aa8375acf44f730df5de1df4202ada2c7a7be4f2
Submitter: Jenkins
Branch: master

commit aa8375acf44f730df5de1df4202ada2c7a7be4f2
Author: Alexandr Nagovitsyn <email address hidden>
Date: Wed Mar 2 17:32:16 2016 +0300

    Add RAM filters and increased the timers in heat tests

    Recently, a few new bugs associated with timers and error
    'failed by timeout on step wait stack CREATE_COMPLETE status'
    Research has shown that an error in most cases occurs due to a
    slow environments.
    Typically, a small amount of RAM (<3 GB).
    Tests autoscaling and wait condition require more than 7 GB
    Also increased the other timers in (+ 120s in verify steps,
    + 30s in create stack steps, +10s in some other.

    Change-Id: I8066581e6358e84655ac3b419dcaa324e4f57aca
    Closes-bug:#1544179
    Closes-bug:#1539721
    Closes-bug:#1525200

Changed in fuel:
status: In Progress → Fix Committed

Done

Reviewed: https://review.openstack.org/285595
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=eb22a9246287c59f40d0f51f26afd24a81b0e66f
Submitter: Jenkins
Branch: master

commit eb22a9246287c59f40d0f51f26afd24a81b0e66f
Author: Andrew Woodward <email address hidden>
Date: Fri Feb 26 16:53:45 2016 -0800

    Drop openstack::heat

    This uplifts the openstack::heat directly into the calling task.

    Closes-bug: #1544179
    Change-Id: Id3a39e0c75832e64c2f60664c0ffc0acfa7c0ec3
    Blueprint: fuel-remove-conflict-openstack

Andrew Woodward (xarses) wrote :

Please confirm if #285595 resolves the problem in CI

Alex Ermolov (aermolov) on 2016-05-23
tags: added: non-release

Ok, the issue was fixed, marked as Fix Released.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: swarm-blocker
Alexey Stupnikov (astupnikov) wrote :

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers