[OSTF] Check stack autoscaling test fails on 'Wait for the 2nd instance to be launched' with reason 'Time limit exceeded'

[Andrey Lavrentyev]

Deployment with 3 controllers, 3 mongo, 1 compute, 1 cinder and NeutronVLAN

Scenario:
1. Deploy environment with default storages and 3 controllers, 3 mongo, 1 compute, 1 cinder and NeutronTUN or NeutronVLAN
2. Shut down 1 Mongo node
3. Verify networks
4. Run OSTF tests <<<<< FAILED HERE
5. Turn on Mongo node
6. Verify networks
7. Run OSTF tests

Actual result:
OSTF autoscale test fail (step #4):

Output from Fuel Web Healthcheck:

Target component: Heat

Scenario:
1. Create test flavor.
2. Create a keypair.
3. Save generated private key to file on Controller node.
4. Create a security group.
5. Create a stack.
6. Wait for the stack status to change to 'CREATE_COMPLETE'.
7. Create a floating IP.
8. Assign the floating IP to the instance of the stack.
9. Wait for instance is ready for load.
10. Load the instance CPU to initiate the stack scaling up.
11. Wait for the 2nd instance to be launched. <<<<< FAILED HERE
12. Release the instance CPU to initiate the stack scaling down.
13. Wait for the 2nd instance to be terminated.
14. Delete the file with private key.
15. Delete the stack.
16. Wait for the stack to be deleted.

Diagnostic Snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B5HPBFb7K7gXMXg0MEdoR09qODQ/view?usp=sharing

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Alexander Nagovitsyn (gluk12189)]

We have problem with fuel ostf heat autoscaling test.
the point is that it is now Ceilometer metric considered every 10 minutes. (Previously, every 1 minute). Therefore now we need 20 minutes for autoscale test (Previously ~5 m)
There are 2 variants:
1) skip this test
2) increase the timers to 20 minutes ( and change CPU_UTILL -this metric will be removed, to CPU metric on template)
There is also variant of to use instance metric, but we get a synthetic test, which is not very good

[Tatyanka (tatyana-leontovich)]

+ 1 for option 2 . This test is in Platform section (as long test), so it is not the problem, but please be sure that it will not affects all service system tests (where we execute platfrom tests)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/274150

[Alexander Nagovitsyn (gluk12189)]

Add to review:
https://review.openstack.org/#/c/274150/

I used option 3 ( metric instance) because cpu metric is not working scale_down policy - This metric counts the number of CPU cycles. It is ever increasing number.

In addition, instance metric reduces test up to 5 minutes.
if the Ceilometer team provides a more appropriate metric, I'll rewrite the test again with new metric.

[Roman Podoliaka (rpodolyaka)]

Removing area-mos as this must only affect the test

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274150
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=e4062f3146fc46109bbe6835dcda1876718f8427
Submitter: Jenkins
Branch: master

commit e4062f3146fc46109bbe6835dcda1876718f8427
Author: Alexandr Nagovitsyn <email address hidden>
Date: Fri Jan 29 19:32:15 2016 +0300

    Update heat autoscaling test and test templates

    Now Ceilometer does not support cpu_util metric. Thats why
    it was needed to rewrite existing test_autoscaling to use another
    metric (network.incoming.bytes).

    Change-Id: I2cfda7624c9736e68d1d2fde4cdc37457cf5a593
    Closes-bug:#1538166

[Timur Nurlygayanov (tnurlygayanov)]

Marked as Critical for 8.0, I ask to make HCF exception for the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275837
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=3bc76a63a9e7d195ff34eadc29552f4235fa6c52
Submitter: Jenkins
Branch: stable/8.0

commit 3bc76a63a9e7d195ff34eadc29552f4235fa6c52
Author: Timur Nurlygayanov <email address hidden>
Date: Wed Feb 3 20:22:46 2016 +0300

    Update heat autoscaling test and test templates

    Now Ceilometer does not support cpu_util metric. Thats why
    it was needed to rewrite existing test_autoscaling to use another
    metric (network.incoming.bytes).

    Change-Id: I2cfda7624c9736e68d1d2fde4cdc37457cf5a593
    Closes-bug:#1538166

[Alexander Nagovitsyn (gluk12189)]

as new fix, i will edit timers for stack creation and other functions
This will avoid problems on a slow environment