Not enough resources to run the test

[Dmitry Belyaninov]

Detailed bug description:
SWARM 7.0

The test should be passed. Current state - skipped with fail message.
Steps to reproduce:
run deploy_murano_ha_one_controller_tun.deploy_murano_ha_one_controller_tun test
Expected results:
Passed
Actual result:
Failed
Reproducibility:
100%
Workaround:
tbu
Impact:
Swarm test failed
Description of the environment:
according to test description
Additional information:
The following tests have not succeeded, while they must have passed: [u'Check that user can deploy application in Murano environment (fuel_health.tests.tests_platform.test_murano_linux.MuranoDeployLinuxServicesTests.test_deploy_dummy_app). Test status: skipped, message: This test requires more hardware resources of your OpenStack cluster: your cloud should allow to create at least 1 VM with 4096 MB of RAM, 40 HDD and 2 vCPUs. You need to remove some resources or add compute nodes to have an ability to run this OSTF test.']

[Dmitry Belyaninov (dbelyaninov)]

https://mirantis.testrail.com/index.php?/tests/view/3771831

console output:
https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.services_ha_one_controller/16/console

[Igor Shishkin (teran)]

Hello Dmitry,

Frankly speaking I can't get what this issue is about.
You have broken test but I can't find any reason from our side at the moment and it's not clear from the job build log.
So please provide more details about which resources are not enough?

[Dmitry Belyaninov (dbelyaninov)]

I suppose this is test issue, maybe we should increase default virtual nodes params.

[Igor Shishkin (teran)]

@Dmitry, here's the thing.
For tests QA team is charged, for infrastructure - DevOps team.
So please clarify which changes you asking us to perform.
At the moment we can't get the reason of the fail so please investigate the issue carefully before failing bug.

[Alexander Nagovitsyn (gluk12189)]

Swarm tests are failed because 2 tests ( wait condition and autoscalong) skipped.
(using raise SkipTest(reason) function)

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha_one_controller/81/console

For the stable operation of these tests have been added to the Ram and CPU filters.
For this tests need more than 7gb RAM and >= 2 CPU
As i can see, swarm lab using >5gb ram

Heat team does not guarantee a stable job at an incorrect test environment. For this reason, we have introduced tests filters.

I see are 2 ways to solve the problem
- increase resource environment
- always expect 2 fails

[Igor Shishkin (teran)]

Assigning to CI team since first option requires JJB configuration update which CI team is responsible for.

[Aleksandra Fedorova (bookwar)]

QA team, please check if this bug is still relevant for 9.0 and 10.0, and send patch to swarm variables if needed

[Tatyanka (tatyana-leontovich)]

move to MOS-QA team for 9.0 and 10.0 according to murano was removed from vanilla fuel and start to deploy as plugin

[Timur Nurlygayanov (tnurlygayanov)]

Hi team,

Murano require compute nodes with free RAM.
Please try to run the same tests with compute nodes with 8GB fee RAM and add more information about your lab configuration to the comments.

Moved to Incomplete because it looks like you are using incorrect lab configuration for now.

Thank you!

[Victor Ryzhenkin (vryzhenkin)]

FYI, Murano wasn't removed from Vanilla 9.0 Fuel. In 10.0 - yes.

[Anastasia Kuznetsova (akuznetsova)]

The same problem for one of the Heat OSTF tests:

AssertionError: The following tests have not succeeded, while they must have passed:
  - Check stack autoscaling (fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_autoscaling). Test status: skipped, message: This test requires more hardware resources of your OpenStack cluster: your cloud should allow to create at least 1 VM with 7000 MB of RAM, 40 HDD and 2 vCPUs. You need to remove some resources or add compute nodes to have an ability to run this OSTF test.

[Alexander Nagovitsyn (gluk12189)]

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha/84/console
Failed

slave-02:
    Host node-1
    Roles:
       Hiera:
           - primary-controller
           - primary-mongo
       Nailgun:
           - controller
           - mongo
    Memory:
       RAM:
          used 5827
          cached 110
          free 142
          shared 56
          total 5969
          buffers 2
       SWAP:
          total 6143
          free 5124
          used 1019

Need total RAM > 7000mb!

[Alexander Nagovitsyn (gluk12189)]

On review
https://review.fuel-infra.org/#/c/20060/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20060
Submitter: Igor Belikov <email address hidden>
Branch: master

Commit: a5504441177d7c546c7ad61f1fbc86a787155901
Author: Alexandr Nagovitsyn <email address hidden>
Date: Mon Apr 25 16:09:28 2016

Fixed service-ha heat tests

Service-ha heat tests (autoscaling and waitcondition)
need more than 7gb ram for stable work.
In the test, we deployed three or more virtual machines
on that we need a small supply of RAM.
This commit fixed this issue.

Change-Id: I6cf84d045b2e8d63d43930c1887a8db739ae96ac
Closes-Bug: #1563765

[Alexander Nagovitsyn (gluk12189)]

 Memory:
       RAM:
          used 1453
          cached 810
          free 5523
          shared 0
          total 6977
          buffers 137

SLAVE_NODE_MEMORY=7168

where ~200mb? I can't understand this magic

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20386
Submitter: Artur Kaszuba <email address hidden>
Branch: master

Commit: ed1543419441ee50eb10007e5bfe791abc721f56
Author: Alexandr Nagovitsyn <email address hidden>
Date: Fri May 6 10:21:20 2016

Fixed service_ha and service_ha_one_controller tests

Service-ha heat tests (autoscaling and waitcondition)
need more than 7gb ram for stable work.
In the test, we deployed three or more virtual machines
on that we need a small supply of RAM.
This commit fixed this issue.
After recearch i see that older ram value= 7168mb in SLAVE_NODE_MEMORY
actually = 6977mb total RAM on real machine.
A certain amount of memory is given to the OS.
We use slaves with 64gb ram for this label
and the jobs run exclusively on those slaves, so there shouldn't be any memory overcommit.
I added +1024mb ram on slaves in this tests groups.

Change-Id: I6e5fb204e325952f3665e1a485e5af0ec87e3f42
Closes-Bug: #1563765

[Mikhail Samoylov (msamoylov)]

Fix not helped:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha/99/console

The same problem with test: "deploy_murano_ha_one_controller_tun" because this test not in group "services_ha" he is in groups: "services", "services.murano", "services_ha_one_controller".

[Timur Nurlygayanov (tnurlygayanov)]

Fixes were merged but tests continue to fail.

[Timur Nurlygayanov (tnurlygayanov)]

Ok, let's wait for one day and verify that fix works.

[Timur Nurlygayanov (tnurlygayanov)]

The issue wasn't reproduced anymore, marked as Fix Released, verified on downstream MOS 9.0 #350:

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha/105/testReport/(root)/deploy_heat_ha/deploy_heat_ha/

So, the test continue to fail but already because of other problem.

[Bogdan Dobrelya (bogdando)]

Closing as anactive