"Advanced stack actions" OSTF test fails on 10.0 + MOS packages

[Dmitry Burmistrov]

Detailed bug description:
 ISO: fuel-10.0-community-507
 OSTF test: "Advanced stack actions: suspend, resume and check"

 Test run status:
    Actual value - SHUTOFF,
    Target component: Heat

 Test scenario:
 1. Create test flavor.
 2. Create a stack.
 3. Wait until the stack status will change to 'CREATE_COMPLETE'.
 4. Call stack suspend action.
 5. Wait until the stack status will change to 'SUSPEND_COMPLETE'.
 6. Check that stack resources are in 'SUSPEND_COMPLETE' status.
 7. Check that server owned by stack is in 'SUSPENDED' status.
 8. Call stack resume action.
 9. Wait until the stack status will change to 'RESUME_COMPLETE'.
 10. Check that stack resources are in 'RESUME_COMPLETE' status.
 11. Check that instance owned by stack is in 'ACTIVE' status.
 12. Call stack check action.
 13. Wait until the stack status will change to 'CHECK_COMPLETE'.
 14. Check that stack resources are in 'CHECK_COMPLETE' status.
 [x] 15. Check that instance owned by stack is in 'ACTIVE' status. # failing here
 16. Delete the stack and wait for the stack to be deleted.

 Total max duration: 900 s.
 Actual time: 159.3 - so it is not caused by timeout

 There are no errors in heat logs:
 root@node-3:~# fgrep -v INFO /var/log/heat/heat-* | fgrep 2016-08-12 | wc -l
 0
 root@node-3:~#

 I don't see information in logs about checks for ACTIVE state:
 [root@nailgun ~]# fgrep fuel_health.heatmanager /var/log/ostf/cluster_1_tests_platform.log | fgrep ACTIVE | wc -l
 0
 [root@nailgun ~]#

 For CHECK_COMPLETE (as an example) there are some rows:
 [root@nailgun ~]# fgrep fuel_health.heatmanager /var/log/ostf/cluster_1_tests_platform.log | fgrep CHECK_COMPLETE | wc -l
 2
 [root@nailgun ~]# fgrep fuel_health.heatmanager /var/log/ostf/cluster_1_tests_platform.log | fgrep CHECK_COMPLETE
 fuel_health.heatmanager: DEBUG: Waiting for <Stack {u'parent': None, u'disable_rollback': True, u'description': u'Template which creates single instance\n', u'parameters': {u'OS::project_id': u'705c636e36354992aaab426f2eda2b04', u'network': u'7b0c55a3-8863-43ab-b2b5-242546222d41', u'OS::stack_name': u'ost1_test-heat-stack-999971657', u'ImageId': u'TestVM', u'OS::stack_id': u'a00ed569-eb52-4fab-b657-f2f435ee4b51', u'InstanceType': u'ost1_test-heat-flavor-334834281'}, u'deletion_time': None, u'stack_name': u'ost1_test-heat-stack-999971657', u'stack_user_project_id': u'6ebfb9bbc1a74ee2a4a251874208dea4', u'stack_status_reason': u'Stack CHECK started', u'creation_time': u'2016-08-12T15:48:04', u'links': [{u'href': u'http://10.21.6.3:8004/v1/705c636e36354992aaab426f2eda2b04/stacks/ost1_test-heat-stack-999971657/a00ed569-eb52-4fab-b657-f2f435ee4b51', u'rel': u'self'}], u'capabilities': [], u'notification_topics': [], u'outputs': [{u'output_value': u'fb04ca78-9d31-4efd-afdb-f5c1f9dac7ee', u'output_key': u'servers', u'description': u'No description given'}], u'timeout_mins': None, u'stack_status': u'CHECK_IN_PROGRESS', u'stack_owner': None, u'updated_time': None, u'id': u'a00ed569-eb52-4fab-b657-f2f435ee4b51', u'tags': None, u'template_description': u'Template which creates single instance\n'}> to get to CHECK_COMPLETE status. Currently in CHECK_IN_PROGRESS status.
 fuel_health.heatmanager: DEBUG: List of fetched objects: [<Resource {u'resource_name': u'Server', u'links': [{u'href': u'http://10.21.6.3:8004/v1/705c636e36354992aaab426f2eda2b04/stacks/ost1_test-heat-stack-999971657/a00ed569-eb52-4fab-b657-f2f435ee4b51/resources/Server', u'rel': u'self'}, {u'href': u'http://10.21.6.3:8004/v1/705c636e36354992aaab426f2eda2b04/stacks/ost1_test-heat-stack-999971657/a00ed569-eb52-4fab-b657-f2f435ee4b51', u'rel': u'stack'}], u'logical_resource_id': u'Server', u'creation_time': u'2016-08-12T15:48:06', u'resource_status': u'CHECK_COMPLETE', u'updated_time': u'2016-08-12T15:48:06', u'required_by': [], u'resource_status_reason': u'state changed', u'physical_resource_id': u'fb04ca78-9d31-4efd-afdb-f5c1f9dac7ee', u'resource_type': u'OS::Nova::Server'}>]
 [root@nailgun ~]#

 I also see logs for 16th step (16. Delete the stack and wait for the stack to be deleted.):
 fuel_health.heatmanager: DEBUG: Deletion of specified stack: a00ed569-eb52-4fab-b657-f2f435ee4b51
 fuel_health.heatmanager: DEBUG: Getting desired stack: a00ed569-eb52-4fab-b657-f2f435ee4b51.
 fuel_health.heatmanager: DEBUG: Getting desired stack: a00ed569-eb52-4fab-b657-f2f435ee4b51.
 fuel_health.heatmanager: DEBUG: Getting desired stack: a00ed569-eb52-4fab-b657-f2f435ee4b51.
 fuel_health.heatmanager: DEBUG: Stack "a00ed569-eb52-4fab-b657-f2f435ee4b51" has been deleted.

 As I understand in case of failure on step #15 step #16 shouldn't be executed at all, so probably there are some issues with OSTF test code.

 heat engine log: http://paste.openstack.org/show/556127/
 heat api log: http://paste.openstack.org/show/556128/
 There are only parts of logs - only for test time.
 Other heat logs contain 200 OK only.

 Test log is attached to this bug.

Steps to reproduce:
 Deploy 1 controller + 1 compute environment on fresh 10.0 ISO with MOS packages. Run OSTF test (Advanced stack actions: suspend, resume and check).
Expected results:
 Test passes.
Actual result:
 Test failed.
Description of the environment:
 Operation system: Ubuntu 16.04

[Alexander Nagovitsyn (gluk12189)]

We cant't reproduse this bug on latest 10.0 (fuel-10.0-community-589-2016-08-26_11-14-00.iso)

[Alexander Nagovitsyn (gluk12189)]

I'm deploy 9.1 claster with 9.0-2016-08-29-152322 repo
(https://product-ci.infra.mirantis.net/view/9.x/job/9.x.main.ubuntu.bvt_2/197/console)

Advanced stack actions: suspend, resume and check is passed

Also see this latest job https://product-ci.infra.mirantis.net/view/9.x_swarm/job/9.x.system_test.ubuntu.services_ha/43/console
:
2016-08-30 03:54:19,494 - INFO fuel_web_client.py:223 -- Passed OSTF test fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_advanced_actions found
2016-08-30 03:54:19,786 - INFO nailgun_client.py:335 -- Get tests finish with success
2016-08-30 03:54:20,238 - INFO fuel_web_client.py:211 -- Assert OSTF tests are passed at cluster #1