OSTF Heat platform tests failed after deploy environment

[Dmitry Tyzhnenko]

3 OSTF test failed after deploy environment.

Steps:
1. Create new environment
2. Choose Neutron, tunnelling segmentation
3. Choose Ceph RBD for ephemeral volumes and uncheck Cinder LVM over iSCSI for volumes, set replication factor to 2
4. Add 5 controllers
5. Add 1 compute
6. Add 2 ceph
7. Change default disks partitioning for ceph nodes for vdc
8. Change public default mask from /24 to /25
9. Verify networks
10. Deploy the environment
11. Run OSTF

Expected result:
  All steps pass

Actual result:
  11 step failed. 3 tests are failed:

    Typical stack actions: create, delete, show details, etc. - Failed on step 2 2. Create a stack.
    Advanced stack actions: suspend, resume and check - Failed on step 2. Create a stack.
    Check stack rollback - Failed on step 4. Wait for the stack to be deleted in result of rollback after
            expiration of timeout defined in WaitHandle resource of the stack.

Example of HealthCheck errors from UI - http://paste.openstack.org/show/485450/

Snapshot log - https://drive.google.com/a/mirantis.com/file/d/0B8U7EvTbuAOlUHJEUWp6a2xObFk/view?usp=sharing

Fuel 8.0-478
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Matthew Mosesohn (raytrac3r)]

Moving to 8.0-updates because it won't land in time for 8.0 HCF

[Maksim Malchuk (mmalchuk)]

this is actually the bug in the OSTF tests:

fuel_health.heatmanager: DEBUG: Getting desired stack: 76d21528-bfdc-4d46-8132-54d783416f1d.
heatclient.common.http: DEBUG: curl -g -i -X GET -H 'X-Auth-Token: {SHA1}8cf990f3340f3fff6f19045c99aa177301e5f420' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' -k http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/76d21528-bfdc-4d46-8132-54
d783416f1d
heatclient.common.http: DEBUG:
HTTP/1.1 302 Found
content-length: 214
via: 1.1 apache_api_proxy:8004
server: Apache
connection: close
location: http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/ost1_test-heat-stack-1797776849/76d21528-bfdc-4d46-8132-54d783416f1d
date: Tue, 02 Feb 2016 16:30:41 GMT
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-802cf7d5-c7e3-4ab0-987d-f49ad6a10ae0

302 Found

The resource was found at http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/ost1_test-heat-stack-1797776849/76d21528-bfdc-4d46-8132-54d783416f1d; you should be redirected automatically.

heatclient.common.http: DEBUG: curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}8cf990f3340f3fff6f19045c99aa177301e5f420' -k http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/ost1_test-heat-stack-1797776849/76d21528-bfdc-4d46-8132-54d783416f1d
fuel_health.common.test_mixins: INFO: Timeout 20s exceeded for stack creation
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 223, in __exit__
    raise AssertionError(msg)
AssertionError: Time limit exceeded while waiting for stack creation to finish.

fuel_health.heatmanager: DEBUG: Deletion of specified stack: 76d21528-bfdc-4d46-8132-54d783416f1d
fuel_health.heatmanager: DEBUG: Getting desired stack: 76d21528-bfdc-4d46-8132-54d783416f1d.
heatclient.common.http: DEBUG: curl -g -i -X GET -H 'X-Auth-Token: {SHA1}8cf990f3340f3fff6f19045c99aa177301e5f420' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' -k http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/76d21528-bfdc-4d46-8132-54d783416f1d
heatclient.common.http: DEBUG:
HTTP/1.1 302 Found
content-length: 214
via: 1.1 apache_api_proxy:8004
server: Apache
connection: close
location: http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/ost1_test-heat-stack-1797776849/76d21528-bfdc-4d46-8132-54d783416f1d
date: Tue, 02 Feb 2016 16:30:46 GMT
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-a272f459-b405-4a4e-826d-4a307b5f4167

302 Found

The resource was found at http://10.109.3.3:8004/v1/4961fdb49ee14c379378304e7e112c12/stacks/ost1_test-heat-stack-1797776849/76d21528-bfdc-4d46-8132-54d783416f1d; you should be redirected automatically.

[Dmitry Tyzhnenko (dtyzhnenko)]

The bug does not reproduce on 8.0-510. Move to invalid.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "510"
  build_id: "510"
  fuel-nailgun_sha: "41170db11c366af5fe04c1c539c11b2e3e388ef9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "ec7e212972ead554f21b52b9e165156665f659df"
  fuel-ostf_sha: "5fe41945c2a49f26c849df1fd46329f6db1ab6b0"
  fuel-mirror_sha: "351d568fa3b3e4dd062054b91d766aa54d379867"
  fuelmenu_sha: "234cb4cbb30fbd2df00f388c28f31606d9cae15f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "94507c5e4dad6d8cfbd8f5d41aa8389d5335990a"

[Dmitry Tyzhnenko (dtyzhnenko)]

Time on time, we have failure of platform test. I think we can increase time in 4 step. When tests runs on heavy loaded host, 20 sec may will not enough to successfully finish. +10 sec may be good for it.

[Timur Nurlygayanov (tnurlygayanov)]

Hi Dmitry, thank you for the RCA!

We will increase timeout in the next release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287229
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=aa8375acf44f730df5de1df4202ada2c7a7be4f2
Submitter: Jenkins
Branch: master

commit aa8375acf44f730df5de1df4202ada2c7a7be4f2
Author: Alexandr Nagovitsyn <email address hidden>
Date: Wed Mar 2 17:32:16 2016 +0300

    Add RAM filters and increased the timers in heat tests

    Recently, a few new bugs associated with timers and error
    'failed by timeout on step wait stack CREATE_COMPLETE status'
    Research has shown that an error in most cases occurs due to a
    slow environments.
    Typically, a small amount of RAM (<3 GB).
    Tests autoscaling and wait condition require more than 7 GB
    Also increased the other timers in (+ 120s in verify steps,
    + 30s in create stack steps, +10s in some other.

    Change-Id: I8066581e6358e84655ac3b419dcaa324e4f57aca
    Closes-bug:#1544179
    Closes-bug:#1539721
    Closes-bug:#1525200

[Timur Nurlygayanov (tnurlygayanov)]

Ok, the issue was fixed, marked as Fix Released.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.